Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scott Anderson [InfluxData] | InfluxDB Tasks – Beyond Downsampling | InfluxDays Virtual Experience NA 2020

65 views

Published on

Scott Anderson [InfluxData] | InfluxDB Tasks – Beyond Downsampling | InfluxDays Virtual Experience NA 2020

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Scott Anderson [InfluxData] | InfluxDB Tasks – Beyond Downsampling | InfluxDays Virtual Experience NA 2020

  1. 1. Scott Anderson Senior Technical Writer & Technical Lead of the Docs team @ InfluxData InfluxDB Tasks Beyond Downsampling
  2. 2. © 2020 InfluxData. All rights reserved. 2 What is an InfluxDB task? A scheduled Flux script
  3. 3. © 2020 InfluxData. All rights reserved. 3 Why tasks? ● Automate repeated operations ● A replacement for Continuous Queries (CQs) in InfluxDB Cloud and InfluxDB 2.0
  4. 4. © 2020 InfluxData. All rights reserved. 4 Downsampling cascade
  5. 5. © 2020 InfluxData. All rights reserved. 5 CREATE CONTINUOUS QUERY "downsample-daily" ON "my-db" BEGIN SELECT mean("example-field") INTO "average-example-measurement" FROM "example-measurement" GROUP BY time(1h) END
  6. 6. © 2020 InfluxData. All rights reserved. 6 from(bucket: "my-bucket") |> range(start: tasks.lastSuccess(orTime: -task.every) |> filter(fn: (r) => r._measurement == "example-m" and r._field == "value" ) |> aggregateWindow(every: 1h, fn: mean()) |> to(bucket: "downsampled-daily", org: "my-org")
  7. 7. © 2020 InfluxData. All rights reserved. 7 Downsampling InfluxDB Template github.com/influxdata/community-templates
  8. 8. © 2020 InfluxData. All rights reserved. 8
  9. 9. © 2020 InfluxData. All rights reserved. 9
  10. 10. and the potential of Flux BEYOND Downsampling
  11. 11. Monitoring & Alerting
  12. 12. © 2020 InfluxData. All rights reserved. 12
  13. 13. © 2020 InfluxData. All rights reserved. 13 import "influxdata/influxdb/monitor" import "influxdata/influxdb/tasks" import "influxdata/influxdb/v1" check = {_check_id: "00000000deadbeef", _check_name: "Average memory monitor", _type: "threshold", tags: {}} task_data = from(bucket: "my-bucket") |> range(start: tasks.lastSuccess(orTime: -task.every), stop: now()) |> filter(fn: (r) => r._measurement == "telegraf" and r._field == "mem") |> mean() ok = (r) => r._value <= 60.0 crit = (r) => r._value > 95.0 warn = (r) => r._value > 80.0 and r._value <= 95.0 info = (r) => r._value > 60.0 and r._value <= 80.0 messageFn = (r) => ("Average mem check: ${r.host}'s average memory usage is ${r._value}%.") task_data |> v1.fieldsAsCols() |> monitor.check(data: check, messageFn: messageFn, ok: ok, crit: crit, warn: warn, info: info) System-generated check task
  14. 14. © 2020 InfluxData. All rights reserved. 14 import "influxdata/influxdb/monitor" import "influxdata/influxdb/tasks" import "influxdata/influxdb/v1" check = {_check_id: "00000000deadbeef", _check_name: "Average memory monitor", _type: "threshold", tags: {}} task_data = from(bucket: "my-bucket") |> range(start: tasks.lastSuccess(orTime: -task.every), stop: now()) |> filter(fn: (r) => r._measurement == "telegraf" and r._field == "mem") |> mean() ok = (r) => r._value <= 60.0 crit = (r) => r._value > 95.0 warn = (r) => r._value > 80.0 and r._value <= 95.0 info = (r) => r._value > 60.0 and r._value <= 80.0 messageFn = (r) => ("Average mem check: ${r.host}'s average memory usage is ${r._value}%.") task_data |> v1.fieldsAsCols() |> monitor.check(data: check, messageFn: messageFn, ok: ok, crit: crit, warn: warn, info: info) System-generated check task
  15. 15. © 2020 InfluxData. All rights reserved. 15 import "influxdata/influxdb/monitor" import "influxdata/influxdb/tasks" import "influxdata/influxdb/v1" check = {_check_id: "00000000deadbeef", _check_name: "Average memory monitor", _type: "threshold", tags: {}} task_data = from(bucket: "my-bucket") |> range(start: tasks.lastSuccess(orTime: -task.every), stop: now()) |> filter(fn: (r) => r._measurement == "telegraf" and r._field == "mem") |> mean() ok = (r) => r._value <= 60.0 crit = (r) => r._value > 95.0 warn = (r) => r._value > 80.0 and r._value <= 95.0 info = (r) => r._value > 60.0 and r._value <= 80.0 messageFn = (r) => ("Average mem check: ${r.host}'s average memory usage is ${r._value}%.") task_data |> v1.fieldsAsCols() |> monitor.check(data: check, messageFn: messageFn, ok: ok, crit: crit, warn: warn, info: info) System-generated check task
  16. 16. © 2020 InfluxData. All rights reserved. 16
  17. 17. © 2020 InfluxData. All rights reserved. 17 import "influxdata/influxdb/monitor" import "influxdata/influxdb/tasks" import "influxdata/influxdb/v1" check = {_check_id: "00000000deadbeef", _check_name: "Average memory monitor", _type: "threshold", tags: {}} task_data = from(bucket: "my-bucket") |> range(start: tasks.lastSuccess(orTime: -task.every), stop: now()) |> filter(fn: (r) => r._measurement == "telegraf" and r._field == "mem") |> mean() ok = (r) => r._value <= 60.0 crit = (r) => r._value > 95.0 warn = (r) => r._value > 80.0 and r._value <= 95.0 info = (r) => r._value > 60.0 and r._value <= 80.0 messageFn = (r) => ("Average mem check: ${r.host}'s average memory usage is ${r._value}%.") task_data |> v1.fieldsAsCols() |> monitor.check(data: check, messageFn: messageFn, ok: ok, crit: crit, warn: warn, info: info) System-generated check task
  18. 18. © 2020 InfluxData. All rights reserved. 18 import "influxdata/influxdb/monitor" import "influxdata/influxdb/tasks" import "influxdata/influxdb/v1" check = {_check_id: "00000000deadbeef", _check_name: "Average memory monitor", _type: "threshold", tags: {}} task_data = from(bucket: "my-bucket") |> range(start: tasks.lastSuccess(orTime: -task.every), stop: now()) |> filter(fn: (r) => r._measurement == "telegraf" and r._field == "mem") |> mean() ok = (r) => r._value <= 60.0 crit = (r) => r._value > 95.0 warn = (r) => r._value > 80.0 and r._value <= 95.0 info = (r) => r._value > 60.0 and r._value <= 80.0 messageFn = (r) => ("Average mem check: ${r.host}'s average memory usage is ${r._value}%.") task_data |> v1.fieldsAsCols() |> monitor.check(data: check, messageFn: messageFn, ok: ok, crit: crit, warn: warn, info: info) System-generated check task
  19. 19. © 2020 InfluxData. All rights reserved. 19
  20. 20. © 2020 InfluxData. All rights reserved. 20 import "influxdata/influxdb/monitor" import "influxdata/influxdb/slack" endpoint = slack.endpoint( url: "https://slack.com/api/chat.postMessage", token: "SLACK_TOKEN") ) monitor.from( start: -task.every, fn: (r) => r._level == "crit" ) |> endpoint(mapFn: (r) => ({ r with channel: "alerts", text: r.message, color: "danger"}))() System-generated notification task
  21. 21. © 2020 InfluxData. All rights reserved. 21 import "influxdata/influxdb/monitor" import "influxdata/influxdb/slack" endpoint = slack.endpoint( url: "https://slack.com/api/chat.postMessage", token: "SLACK_TOKEN") ) monitor.from( start: -task.every, fn: (r) => r._level == "crit" ) |> endpoint(mapFn: (r) => ({ r with channel: "alerts", text: r.message, color: "danger"}))() System-generated notification task
  22. 22. © 2020 InfluxData. All rights reserved. 22 import "influxdata/influxdb/monitor" import "influxdata/influxdb/slack" endpoint = slack.endpoint( url: "https://slack.com/api/chat.postMessage", token: "SLACK_TOKEN") ) monitor.from( start: -task.every, fn: (r) => r._level == "crit" ) |> endpoint(mapFn: (r) => ({ r with channel: "alerts", text: r.message, color: "danger"}))() System-generated notification task
  23. 23. © 2020 InfluxData. All rights reserved. 23 *
  24. 24. Simple Custom Alerts Send alerts without using InfluxDB checks & notifications
  25. 25. © 2020 InfluxData. All rights reserved. 25 import "influxata/influxdb/tasks" import "influxata/influxdb/slack" import anomaly "contrib/anaisdg/anomalydetection" toSlack = slack.endpoint( url: "https://slack.com/api/chat.postMessage", token: secrets.get(key: "SLACK_TOKEN") ) from(bucket: "example") |> range(start: tasks.lastSuccess(orTime: -task.every)) |> filter(fn: (r) => r._measurement == "m") |> anomaly.mad() |> filter(fn: (r) => r.level == "anomaly") |> toSlack( mapFn: (r) => { r with channel: "anomalies", text: "An anomaly has been detected on ${r.host}. ${r._field} is ${r._value}", color: "danger" } )() Custom anomaly detection alert
  26. 26. © 2020 InfluxData. All rights reserved. 26 import "influxata/influxdb/tasks" import "influxata/influxdb/slack" import anomaly "contrib/anaisdg/anomalydetection" toSlack = slack.endpoint( url: "https://slack.com/api/chat.postMessage", token: secrets.get(key: "SLACK_TOKEN") ) from(bucket: "example") |> range(start: tasks.lastSuccess(orTime: -task.every)) |> filter(fn: (r) => r._measurement == "m") |> anomaly.mad() |> filter(fn: (r) => r.level == "anomaly") |> toSlack( mapFn: (r) => { r with channel: "anomalies", text: "An anomaly has been detected on ${r.host}. ${r._field} is ${r._value}", color: "danger" } )() Custom anomaly detection alert
  27. 27. © 2020 InfluxData. All rights reserved. 27 import "influxata/influxdb/tasks" import "influxata/influxdb/slack" import anomaly "contrib/anaisdg/anomalydetection" toSlack = slack.endpoint( url: "https://slack.com/api/chat.postMessage", token: secrets.get(key: "SLACK_TOKEN") ) from(bucket: "example") |> range(start: tasks.lastSuccess(orTime: -task.every)) |> filter(fn: (r) => r._measurement == "m") |> anomaly.mad() |> filter(fn: (r) => r.level == "anomaly") |> toSlack( mapFn: (r) => { r with channel: "anomalies", text: "An anomaly has been detected on ${r.host}. ${r._field} is ${r._value}", color: "danger" } )() Custom anomaly detection alert
  28. 28. © 2020 InfluxData. All rights reserved. 28 import "influxata/influxdb/tasks" import "influxata/influxdb/slack" import anomaly "contrib/anaisdg/anomalydetection" toSlack = slack.endpoint( url: "https://slack.com/api/chat.postMessage", token: secrets.get(key: "SLACK_TOKEN") ) from(bucket: "example") |> range(start: tasks.lastSuccess(orTime: -task.every)) |> filter(fn: (r) => r._measurement == "m") |> anomaly.mad() |> filter(fn: (r) => r.level == "anomaly") |> toSlack( mapFn: (r) => { r with channel: "anomalies", text: "An anomaly has been detected on ${r.host}. ${r._field} is ${r._value}", color: "danger" } )() Custom anomaly detection alert
  29. 29. © 2020 InfluxData. All rights reserved. 29 import "influxata/influxdb/tasks" import "influxata/influxdb/slack" import anomaly "contrib/anaisdg/anomalydetection" toSlack = slack.endpoint( url: "https://slack.com/api/chat.postMessage", token: secrets.get(key: "SLACK_TOKEN") ) from(bucket: "example") |> range(start: tasks.lastSuccess(orTime: -task.every)) |> filter(fn: (r) => r._measurement == "m") |> anomaly.mad() |> filter(fn: (r) => r.level == "anomaly") |> toSlack( mapFn: (r) => { r with channel: "anomalies", text: "An anomaly has been detected on ${r.host}. ${r._field} is ${r._value}", color: "danger" } )() Custom anomaly detection alert
  30. 30. © 2020 InfluxData. All rights reserved. 30 import "influxata/influxdb/tasks" import "influxata/influxdb/slack" import anomaly "contrib/anaisdg/anomalydetection" toSlack = slack.endpoint( url: "https://slack.com/api/chat.postMessage", token: secrets.get(key: "SLACK_TOKEN") ) from(bucket: "example") |> range(start: tasks.lastSuccess(orTime: -task.every)) |> filter(fn: (r) => r._measurement == "m") |> anomaly.mad() |> filter(fn: (r) => r.level == "anomaly") |> toSlack( mapFn: (r) => { r with channel: "anomalies", text: "An anomaly has been detected on ${r.host}. ${r._field} is ${r._value}", color: "danger" } )() Custom anomaly detection alert
  31. 31. © 2020 InfluxData. All rights reserved. 31 import "influxata/influxdb/tasks" import "influxata/influxdb/slack" import anomaly "contrib/anaisdg/anomalydetection" toSlack = slack.endpoint( url: "https://slack.com/api/chat.postMessage", token: secrets.get(key: "SLACK_TOKEN") ) from(bucket: "example") |> range(start: tasks.lastSuccess(orTime: -task.every)) |> filter(fn: (r) => r._measurement == "m") |> anomaly.mad() |> filter(fn: (r) => r.level == "anomaly") |> toSlack( mapFn: (r) => { r with channel: "anomalies", text: "An anomaly has been detected on ${r.host}. ${r._field} is ${r._value}", color: "danger" } )() Custom anomaly detection alert Execute generated function
  32. 32. Data Ingest
  33. 33. © 2020 InfluxData. All rights reserved. 33 import "csv" csvData = "#datatype,string,long,dateTime:RFC3339,string,double #group,false,false,false,true,false #default,,,,, ,result,table,_time,region,_value ,,0,2018-05-08T20:50:00Z,east,15.43 ,,0,2018-05-08T20:50:20Z,east,59.25 ,,1,2018-05-08T20:50:00Z,west,62.73 " csv.from(csv:csvData) |> to(org: "my-org", bucket: "my-bucket") Write data using annotated CSV
  34. 34. © 2020 InfluxData. All rights reserved. 34 import "experimental/csv" csv.from(url: "https://influx-testdata.s3.amazonaws.com/noaa.csv") |> to(org: "my-org", bucket: "noaa") Write data using remote annotated CSV
  35. 35. © 2020 InfluxData. All rights reserved. 35 import "experimental/array" data = [ {_time: 2018-05-08T20:50:00Z, region: "east", _value: 15.43}, {_time: 2018-05-08T20:50:20Z, region: "east", _value: 59.25}, {_time: 2018-05-08T20:50:00Z, region: "west", _value: 62.73} ] array.from(rows: data) |> to(org: "my-org", bucket: "my-bucket") Write data with using an array of records
  36. 36. © 2020 InfluxData. All rights reserved. 36 // ... response = http.get( url: "http://localhost:8086/health", headers: {Authorization: "Token mYsuP3rS3cRe7T0kEn"} ) responseBody = response.body seedRow = [{_time: now()}] array.from(rows: seedRow) |> map(fn: (r) => { body = json.parse(data: responseBody) return { r with _measurement: "monitoring", _field: "influxdb_status", _value: body.message, name: body.name } }) |> to(org: "my-org", bucket: "my-bucket") Write data with using an HTTP response
  37. 37. © 2020 InfluxData. All rights reserved. 37 // ... response = http.get( url: "http://localhost:8086/health", headers: {Authorization: "Token mYsuP3rS3cRe7T0kEn"} ) responseBody = response.body seedRow = [{_time: now()}] array.from(rows: seedRow) |> map(fn: (r) => { body = json.parse(data: responseBody) return { r with _measurement: "monitoring", _field: "influxdb_status", _value: body.message, name: body.name } }) |> to(org: "my-org", bucket: "my-bucket") Write data with using an HTTP response
  38. 38. © 2020 InfluxData. All rights reserved. 38 // ... response = http.get( url: "http://localhost:8086/health", headers: {Authorization: "Token mYsuP3rS3cRe7T0kEn"} ) responseBody = response.body seedRow = [{_time: now()}] array.from(rows: seedRow) |> map(fn: (r) => { body = json.parse(data: responseBody) return { r with _measurement: "monitoring", _field: "influxdb_status", _value: body.message, name: body.name } }) |> to(org: "my-org", bucket: "my-bucket") Write data with using an HTTP response
  39. 39. © 2020 InfluxData. All rights reserved. 39
  40. 40. © 2020 InfluxData. All rights reserved. 40 import "experimental/prometheus" prometheus.scrape(url: "http://localhost:8086/metrics") |> to( org: "my-org", bucket: "my-bucket" ) Scrape data from a Prometheus endpoint
  41. 41. © 2020 InfluxData. All rights reserved. 41 import "influxdata/influxdb/secrets" import "sql" username = secrets.get(key: "POSTGRES_USER") password = secrets.get(key: "POSTGRES_PASS") sql.from( driverName: "postgres", dataSourceName: "postgresql://${username}:${password}@localhost", query:"SELECT * FROM example_table" ) Query a SQL datasource
  42. 42. Data Enrichment
  43. 43. © 2020 InfluxData. All rights reserved. 43 join()
  44. 44. © 2020 InfluxData. All rights reserved. 44 _time sensorID _field _value 2020-06-01T00:12:00Z TM02001 temp 70.32 2020-06-01T00:12:01Z TM02002 temp 70.45 2020-06-01T00:12:02Z TM02003 temp 98.41 sensorID location status TM02001 SF1.RM406 OK TM02002 SF1.RM412 OK TM02003 SF2.RM290 Requires service
  45. 45. © 2020 InfluxData. All rights reserved. 45 import "sql" import "influxdata/influxdb/secrets" username = secrets.get(key: "POSTGRES_USER") password = secrets.get(key: "POSTGRES_PASS") telemetry = from(bucket: "sensor-data") |> range(start: -task.every) |> filter(fn: (r) => r._measurement == "sensor-data" and r._field == "temp") assets = sql.from( driverName: "postgres", dataSourceName: "postgresql://${username}:${password}@localhost", query:"SELECT * FROM temp-sensors" ) join(tables: {data: telemetry, assets: assets}, on: ["sensorID"]) |> to(org: "my-org", bucket: "enriched-sensor-data") Enrich data with external data
  46. 46. © 2020 InfluxData. All rights reserved. 46 import "sql" import "influxdata/influxdb/secrets" username = secrets.get(key: "POSTGRES_USER") password = secrets.get(key: "POSTGRES_PASS") telemetry = from(bucket: "sensor-data") |> range(start: -task.every) |> filter(fn: (r) => r._measurement == "sensor-data" and r._field == "temp") assets = sql.from( driverName: "postgres", dataSourceName: "postgresql://${username}:${password}@localhost", query:"SELECT * FROM temp-sensors" ) join(tables: {data: telemetry, assets: assets}, on: ["sensorID"]) |> to(org: "my-org", bucket: "enriched-sensor-data") Enrich data with external data
  47. 47. © 2020 InfluxData. All rights reserved. 47 import "sql" import "influxdata/influxdb/secrets" username = secrets.get(key: "POSTGRES_USER") password = secrets.get(key: "POSTGRES_PASS") telemetry = from(bucket: "sensor-data") |> range(start: -task.every) |> filter(fn: (r) => r._measurement == "sensor-data" and r._field == "temp") assets = sql.from( driverName: "postgres", dataSourceName: "postgresql://${username}:${password}@localhost", query:"SELECT * FROM temp-sensors" ) join(tables: {data: telemetry, assets: assets}, on: ["sensorID"]) |> to(org: "my-org", bucket: "enriched-sensor-data") Enrich data with external data
  48. 48. © 2020 InfluxData. All rights reserved. 48 import "sql" import "influxdata/influxdb/secrets" username = secrets.get(key: "POSTGRES_USER") password = secrets.get(key: "POSTGRES_PASS") telemetry = from(bucket: "sensor-data") |> range(start: -task.every) |> filter(fn: (r) => r._measurement == "sensor-data" and r._field == "temp") assets = sql.from( driverName: "postgres", dataSourceName: "postgresql://${username}:${password}@localhost", query:"SELECT * FROM temp-sensors" ) join(tables: {data: telemetry, assets: assets}, on: ["sensorID"]) |> to(org: "my-org", bucket: "enriched-sensor-data") Enrich data with external data
  49. 49. © 2020 InfluxData. All rights reserved. 49 import "sql" import "influxdata/influxdb/secrets" username = secrets.get(key: "POSTGRES_USER") password = secrets.get(key: "POSTGRES_PASS") telemetry = from(bucket: "sensor-data") |> range(start: -task.every) |> filter(fn: (r) => r._measurement == "sensor-data" and r._field == "temp") assets = sql.from( driverName: "postgres", dataSourceName: "postgresql://${username}:${password}@localhost", query:"SELECT * FROM temp-sensors" ) join(tables: {data: telemetry, assets: assets}, on: ["sensorID"]) |> to(org: "my-org", bucket: "enriched-sensor-data") Enrich data with external data
  50. 50. © 2020 InfluxData. All rights reserved. 50 _time sensorID location status _field _value 2020-06-01T00:12:00Z TM02001 SF1 .RM406 OK temp 70.32 2020-06-01T00:12:01Z TM02002 SF1.RM412 OK temp 70.45 2020-06-01T00:12:02Z TM02003 SF2.RM290 Requires service temp 98.41
  51. 51. Data Transfer
  52. 52. © 2020 InfluxData. All rights reserved. 52
  53. 53. © 2020 InfluxData. All rights reserved. 53 import "influxdata/influxdb/secrets" import "influxdata/influxdb/tasks" cloudToken = secrets.get(key: "INFLUXDB_CLOUD_TOKEN") from(bucket: "telemetry") |> range(start: tasks.lastSuccess(orTime: -task.every)) |> aggregateWindow(every: 5m, fn: mean) |> to( host: "https://us-west-2-1.aws.cloud2.influxdata.com", org: "my-cloud-org", bucket: "fleet", token: cloudToken, )
  54. 54. © 2020 InfluxData. All rights reserved. 54
  55. 55. © 2020 InfluxData. All rights reserved. 55
  56. 56. © 2020 InfluxData. All rights reserved. 56
  57. 57. © 2020 InfluxData. All rights reserved. 57
  58. 58. © 2020 InfluxData. All rights reserved. 58 Other Use Cases ● Data sanitization ● Machine learning ● Reports ● Event triggers ● Data generation ● and much more!!
  59. 59. © 2020 InfluxData. All rights reserved. 59 Thank you! Time for some Q&A

×