More Related Content
Similar to [初音ミク] Kinesis でフリーザを撃て! (8)
[初音ミク] Kinesis でフリーザを撃て!
- 27. Tweet を Stream に入れる
Kinesis
Stream
Twitter4J Kinesis MeCab Dynamo Unity
TwitterStream twitterStream = new TwitterStreamFactory().getInstance();
StatusListener listener = new StatusListener() {
@Override
public void onStatus(Status status) {
String key = RandomStringUtils.randomAlphanumeric(10);
String json = DataObjectFactory.getRawJSON(status);
PutRecordRequest putRecordRequest = new PutRecordRequest();
putRecordRequest.setStreamName(PropUtil.getInstance().getProp("StreamName"));
putRecordRequest.setData(ByteBuffer.wrap(json.getBytes(Charset.forName("UTF-8")));
putRecordRequest.setPartitionKey(key);
PutRecordResult putRecordResult = kinesis.putRecord(putRecordRequest);
}
};
twitterStream.addListener(listener);
FilterQuery filterQuery = new FilterQuery();
filterQuery.track(new String[] { PropUtil.getInstance().getProp("Keyword1"),
PropUtil.getInstance().getProp("Keyword2") });
twitterStream.filter(filterQuery);
- 28. Tweet を Stream から出す
Kinesis
Stream
Twitter4J Kinesis MeCab Dynamo Unity
private void processRecordsWithRetries(List<Record> records) {
for (Record record : records) {
boolean processedSuccessfully = false;
String data = null;
for (int i = 0; i < NUM_RETRIES; i++) {
try {
// Get Data
data = decoder.decode(record.getData()).toString();
Status status = DataObjectFactory.createStatus(data );
// MeCab something
// ・・・・・・
processedSuccessfully = true;
break;
} catch (Throwable t) {}
}
}
}
- 29. Tweet を分析して数値化
Kinesis
Stream
Twitter4J Kinesis MeCab Dynamo Unity
Tagger tagger = new Tagger("-Ochasen");
tagger.parse(status.getText().replaceAll("¥n", ""));
Node node = tagger.parseToNode(status.getText().replaceAll("¥n", ""));
Double sum = 0.0;
int count = 0;
for (; node != null; node = node.getNext()) {
String[] mecabArray = node.getFeature().split(",");
String originalWord = node.getSurface();
if (!mecabArray[6].equals("*")) {
originalWord = mecabArray[6]; // 基本形
}
String feature = mecabArray[0];
if (feature.equals("名詞") || feature.equals("動詞")
|| feature.equals("形容詞”) || feature.equals("形容動詞")
|| feature.equals("副詞")) {
// Positive or Negative
sum+=PositiveNegativeJudge.getInstance().getDouble(originalWord+":"+feature);
count++;
}
}
Double point = Math.floor((sum / count + 1.0) * 100.0 / 2.0);
- 30. DynamoDB で集計
Kinesis
Stream
Twitter4J Kinesis MeCab Dynamo Unity
private static PutItemRequest createPutItemRequest(String id, String datetime, String name,
String screenName, String text, Double value) {
Map<String, AttributeValue> putItems = new HashMap<String, AttributeValue>();
putItems.put("AppName", new
AttributeValue().withS(PropUtil.getInstance().getProp("ApplicationName")));
putItems.put("Datetime/SerialNumber", new AttributeValue().withS(datetime+"/"+id));
putItems.put("ScreenName", new AttributeValue().withS("@" + screenName));
putItems.put("Name", new AttributeValue().withS(name));
putItems.put("Text", new AttributeValue().withS(text));
putItems.put("Value", new AttributeValue().withN(Double.toString(value)));
PutItemRequest putItemRequest = new
PutItemRequest().withTableName(PropUtil.getInstance().getProp("DynamoDbTableNameDetail")).withIte
m(putItems);
return putItemRequest;
}
- 51. おまけ(LTの裏側)
[発表直前]
DyanamoDBからデータを取り出すプロセスが高頻度で死亡する事案が発生
ec2-user 14685 0.1 4.2 1625848 71584 ? Sl 02:10 0:27 java KinesisPutter
ec2-user 18622 2.2 4.8 1787796 82660 pts/1 Sl 06:01 0:21 java -Djava.library.path=.:/usr/local/bin/mecab-java KinesisGetter
ec2-user 18944 22.0 4.2 1618412 71836 ? Sl 06:16 0:08 java DynamoJsonCreator
↓
ec2-user 14685 0.1 4.2 1625848 71584 ? Sl 02:10 0:27 java KinesisPutter
ec2-user 18622 2.2 4.9 1839496 83452 pts/1 Sl 06:01 0:21 java -Djava.library.path=.:/usr/local/bin/mecab-java KinesisGetter
cronでプロセスを再起動するように泥臭い対応をしていましたが
もし運悪くプロセス死亡時にデモをすると、リアルタイムに最新データを
取り出せない可能性がありました。困った・・・
*/2 * * * * bash /home/ec2-user/KinesisTwitter/check_putter.sh > /home/ec2-user/cron.log.putter.txt 2>&1
*/2 * * * * bash /home/ec2-user/KinesisTwitter/check_getter.sh > /home/ec2-user/cron.log.getter.txt 2>&1
*/2 * * * * bash /home/ec2-user/KinesisTwitter/check_create_json.sh > /home/ec2-user/cron.log.create.json.txt 2>&1