Description: VIPER, developed by OTICS Advanced Analytics, Inc., is a binary connector for getting data into and out of Apache Kafka. It is integrated with auto machine learning technology called MAADS and HPDE, that allows users to perform Transactional Machine Learning (TML), predictive analytics and optimization on data streams. It can also join data streams to construct a training dataset in real-time for TML. It also allows users to manage algorithms and insights to control egress and ingress as well as giving visibility on who is producing to and consuming from algorithms. \nIt can run in any platform (Linux, Windows, Solaris, etc.), and is compatible with local or cloud versions of Kafka and supports SSL/TLS encryption..
Read the Confluent blog on Transactional Machine Learning for more information.
Please check Github for latest VIPER, HPDE, and VIPERviz versions.
Forthcoming Book: "Transactional Machine Learning with Data Streams and AutoML" (see Github for details)
Two ways to access MAADS-VIPER:
REST API
MAADS Python Library
Start Command: [VIPER system exectuable] [host] [port]
Host can be any host such as "localhost"
Port can be any port on host
SSL/TLS is supported
VIPER will check the current folder its running in for a valid Token.Tok file, you should have received this file from OTICS. If not, or if your Token has expired, email: [email protected]
VIPER REST API:
listtopics, which will list all topics in the Kafka broker
deactivatetopics, which will deactivate, but not delete, the topic so producers cannot produce to the topic and users cannot consume from the topic
activatetopics, which will activate the topic so producers can produce to the topic and users can consume from the topic
createtopics, administrators can create topics in the Kafka broker. Topics can also be created by pointing to the MAADS algorithm servers, VIPER will auto-generate the topics and generate a producer id for each topic. A producer id is need to write to a topic.
viperstats, which will produce all the information on producers, topics, and consumers.
subscribeconsumer, administrators or users can subscribe consumers to a topic/algorithm, VIPER will generate a consumer id needed to consume.
unsubscribeconsumer, administrators can unsubscribe consumers from a topic
producetotopic, administrators can produce to any topic in the Kafka broker. These are predictions or optimal values from algorithms.
consumefromtopic, users can consume from any topic in the Kafka broker
createconsumergroup, creates a consumer group
consumergroupconsumefromtopic, users in a consumer group can consumer from any topic
modifytopicdetails, will modify topic details
modifyconsumerdetails, will modify consumer details
viperhpdepredict, will generate high speed predictions
viperhpdeoptimize, will find optimal values for the independent variables
viperhpdetraining, will do Real-Time Machine Learning (RTML) on streaming data to find the best or optimal algorithm
deactivategroup, which will deactivate, but not delete, a group so producers cannot produce to the group and users cannot consume from the group
activategroup, which will activate the group so producers can produce to the group and users can consume from the group
producetotopicstream, which will produce to multiple topics at the same time
createtrainingdata, which will create a training data set to Real-Time Machine Learning
consumefromstreamtopic, which will consume from a topic made up of multiple topics
createjointopicstreams, which will create a topic made up of multiple topics
vipermaadsoptimize, which will connect to MAADS to execute optimization routine
VIPER REST API Syntax:
listtopics: http://[viperhost]:[viperport]/listtopics?brokerhost=[kafka broker host address]&brokerport=[kafka broker port]&vipertoken=[ADMIN or USER token]&enabletls=[1=TLS enabled, or 0=not enabled]
RETURNS: A JSON string of list of Kafka topics
deactivatetopics: http://[viperhost]:[viperport]/deactivatetopics?topic=[Kafka topic]&vipertoken=[ADMIN token]
RETURNS: None
activatetopics: http://[viperhost]:[viperport]/activatetopics?topic=[Kafka topic]&vipertoken=[ADMIN token]
RETURNS: None
createtopics: http://[viperhost]:[viperport]/createtopics?topic=[Kafka topic]&brokerhost=[kafka broker host address]&brokerport=[kafka broker port]&vipertoken=[ADMIN token]&maadsalgoserver=[http url]&maadsport=[maads algoserver port]&maadsmicroservice=[maads microservice]&maadstoken=[maads token]&replicationfactor=[replication number]&numpartitions=[number of partitions]&companyname=[your company name]&contactname=[name of contact for topic]&contactemail=[contact email]&description=[short description of the topic]&dependentvariable=[Dependent variable]&independentvariables=[list of independentvariables separated by commas - this list MUST match the order in the algorithm]&enabletls=[1=TLS enabled,0=not enabled]
RETURNS: JSON string of the created topics with associated PRODUCER ID. A valid producer id must be used when writing to a topic.
NOTE:
If you enter a maadsalgoserver and port, then VIPER will connect with maads server and auto-create all of the optimal algorithms from this server. This could be convenient if you have lots of algorithms.
You can create custom topics by entering TOPIC
All fields are required, but if TOPIC is empty, then maadsalgoserver, maadsport, maadsmicroservice, maadstoken must be given, otherwise they can be blank.
viperstats: http://[viperhost]:[viperport]/viperstats?brokerhost=[kafka broker host address]&brokerport=[kafka broker port]&vipertoken=[ADMIN token]
RETURNS: A JSON string containing all consumers, producers, size of kafka folder, etc..
NOTE:
These stats will be very important for ADMINs to manage their Kafka brokers. ADMINS can use these tats in an ADMIN console to manage the entire publishing, and consuming of insights from algorithms across their entire enterprise.
subscribeconsumer: http://[viperhost]:[viperport]/subscribeconsumer?brokerhost=[kafka broker host address]&brokerport=[kafka broker port]&vipertoken=[ADMIN or USER token]&companyname=[your company name]&contactname=[your contact name]&contactemail=[your contact email]&location=[your location]&topic=[Topic you want to subscribe to]
RETURNS: A JSON string containing a CONSUMER ID, a consumer id must be used to consume from a topic.
unsubscribeconsumer: http://[viperhost]:[viperport]/unsubscribeconsumer?brokerhost=[kafka broker host address]&brokerport=[kafka broker port]&vipertoken=[ADMIN token]&consumerid=[a valid consumer id]
RETURNS: None
producetotopic: http://[viperhost]:[viperport]/producetotopic?topic=[valid topic]&maadsalgokey=[MAADS algorithm key]&vipertoken=[ADMIN token]&producerid=[a valid producer id, generated when topic was created]&inputdata=[input data for the MAADS algorithm]&getoptimal=[1 or 0 - if you want to retrieve optimal values]&externalprediction=[your own prediction number]&maadstoken=[a valid maads token]&enabletls=[1=TLS enables,0=not enabled]&delay=[Time in milliseconds before VIPER backsout from writing messages]
RETURNS: A write to the Kafka topic for consumers to consume.
NOTES:
If you have your own machine learning model, you can generate a prediction from that model and send it to VIPER to publish by putting it in the externalprediction. Meaning you do not have to use the MAADS model.
If you use the MAADS optimal algorithm, inputdata must be specified.
You can even get optimal values from MAADS algorithms by setting getoptimal=1, this assume you used the MAADS optimize function from the MAADS python library.
If you are using MAADS Hyperpredictions and passing INPUTDATA then you need to URL encode the commas.
consumefromtopic: http://[viperhost]:[viperport]/consumefromtopic?topic=[valid topic]&offset=[a number that represents where to start the reading, initial value is 0]&partition=[kafka partition, or -1 to auto detect partition]&protocol=[set to tcp]&vipertoken=[ADMIN or USER token]&consumerid=[a valid consumer id]&companyname=[your company name]&enabletls=[1=TLS enabled,0=TLS not enable]&delay=[Time in milliseconds before VIPER backsout from Reading messages]
RETURNS: A JSON string of the values consumed and the last offset.
NOTES:
Initial OFFSET should be 0, subsequent calls must use the offset
You can use the offset to 'ROLLBACK' predictions from the beginning - in case you missed them.
consumergroupconsumefromtopic: http://[viperhost]:[viperport]/consumergroupconsumefromtopic?topic=[valid topic]&offset=[a number that represents where to start the reading, initial value is 0]&partition=[kafka partition, or -1 to autodetect]&protocol=[set to tcp]&vipertoken=[ADMIN or USER token]&consumerid=[a valid consumer id]&companyname=[your company name]&groupid=[a group id]&enabletls=[1=TLS enables,0=not enabled]&delay=[Time in milliseconds before VIPER backsout from Reading messages]
RETURNS: A JSON string of the values consumed and the last offset.
NOTES:
Initial OFFSET should be 0, subsequent calls must use the offset
You can use the offset to 'ROLLBACK' predictions from the beginning - in case you missed them.
modifytopicdetails: http://[viperhost]:[viperport]/modifytopicdetails?brokerhost=[kafka broker host]&brokerport=[kafka broker port]&topic=[valid topic]&companyname=[your company name]&contactname=[name of contact for topic]&contactemail=[contact email]&description=[short description of the topic]&isgroup=[1 if the topic is a group, 0 otherwise]
RETURNS: None
modifyconsumerdetails: http://[viperhost]:[viperport]/modifyconsumerdetails?brokerhost=[kafka broker host]&brokerport=[kafka broker port]&consumerid=[valid consumerid]&companyname=[your company name]&contactname=[name of contact for topic]&contactemail=[contact email]&description=[short description of the topic]
RETURNS: None
viperhpdepredict: http://[viperhost]:[port]/viperhpdepredict?consumefrom=[topic to consume from]&produceto=[topic to produce to]&timeout=[timeout for connection to hpde]&vipertoken=[valid token]&companyname=[company name]&brokerhost=[kafka broker]&brokerport=[kafka port]&consumerid=[valid consumer id]&producerid=[valid producer id]&hpdehost=[hpde host]&hpdeport=[hpde port]&enabletls=[1=TLS enables,0=not enabled]&delay=[Time in milliseconds before VIPER backsout from Reading messages]
RETURNS: JSON object of the prediction
viperhpdeoptimize: http://[viperhost]:[port]/viperhpdeoptimize?consumefrom=[topic to consume from]&produceto=[topic to produce to]&timeout=[timeout for HPDE connection]&vipertoken=[valid viper token]&companyname=[company name] &brokerhost=[kafka broker host]&brokerport=[kafka broker port]&consumerid=[consumer id]&producerid=[producer id]&hpdehost=[HPDE host]&hpdeport=[HPDE port]&enabletls=[1=TLS enables,0=not enabled]&delay=[Time in milliseconds before VIPER backsout from Reading messages]
RETURNS: JSON object of the optimal values
viperhpdetraining: http://[viperhost]:[port]/viperhpdetraining?consumefrom=[topic to consume from]&produceto=[topic to produce to]&timeout=[timeout for HPDE connection]&vipertoken=[valid viper token]&offset=[kafka offset]&companyname=[company name]&brokerhost=[kafka broker host]&brokerport=[kafka port]&consumerid=[valid consumerid]&producerid=[valid producer id]&hpdehost=[HPDE host]&hpdeport=[HPDE port]&enabletls=[1=TLS enables,0=not enabled]&delay=[Time in milliseconds before VIPER backsout from Reading messages]&viperconfigfile=[path to VIPER.ENV file needed if enabletls=1]&offset=[offset to read from, -1 to read from last commit]&modelruns=[number of iterations]
RETURNS: JSON object of optimal algorithm
deactivategroup: http://[viperhost]:[port]/deactivategroup?topic=[kafka topic]&vipertoken=[Admin token]
RETURN: None
activategroup: http://[viperhost]:[port]/activategroup?topic=[kafka topic]&vipertoken=[Admin token]
RETURN: None
vipermaadsoptimize: http://[viperhost]:[viperport]/vipermaadsoptimize?hyperoptimize=1&ismin=[1,0]&algo1=[algo1]&algo2=[algo2]&algo3=[algo3]&perc=[min_[0-100]_max_[0-100]]&a1cons=[constraints]&a2cons=[a2 constrainttype]&a3cons=[a3 constraint type]&rowlen=[number of iterations]&step=[1,2,3]&objeq=[1 or -1]&forceupdate=[1 or 0]&blimits=[var:step:min:max:[integer or float]]&maadstoken=[maadstoken]&enabletls=[1=TLS enabled,0=not enabled]&delay=[Time in milliseconds before VIPER backsout from writing messages]
RETURN: JSON object
producetotopicstream:
http://[viperhost]:[viperport]/producetotopicstream?topicname=[topic name of joined topics]&producerid=[ProducerId]&offset=[offset]&vipertoken=[vipertoken]&enabletls=[1=TLS enabled,0=not enabled]&delay=[Time in milliseconds before VIPER backsout from Reading messages]
RETURN: JSON object
createtrainingdata: http://[viperhost]:[viperport]/createtrainingdata?consumefrom=[topic name of joined topics]&produceto=[Topic to produce to]&vipertoken=[viper token]&dependentvariable=[dependentvariable]&independentvariables=[independent variables separated by comma]&offset=[offset]&companyname=[company name]&consumerid=[consumer id] &producerid=[producer id]&enabletls=[1=TLS enabled,0=not enabled]&delay=[Time in milliseconds before VIPER backsout from Reading messages]
RETURN: JSON object
consumefromstreamtopic:
http://[viperhost]:[viperport]/consumefromstreamtopic?topic=[topic of joined topics]&consumerid=[consumerid]&offset=[offset]&companyname=[company name]&vipertoken=[viper token]&enabletls=[1=TLS enabled,0=not enabled]&delay=[Time in milliseconds before VIPER backsout from Reading messages]&partition=[partition number, or -1 to autodetect]
RETURN: JSON object
createconsumergroup: http://[viperhost]:[viperport]/createconsumergroup?topic=[topic, separate by commas for multiple topics]&groupname=[group name]&vipertoken=[vipertoken]&contactname=[contact name]&contactemail=[contact email]&location=[location]&companyname=[company name]&description=[topic desxription]&enabletls=[1=TLS enabled,0=not enabled]
RETURNS: JSON object
createjointopicstreams:
http://[viperhost]:[viperport]/createjointopicstreams?topicname=[topic name]&topicstojoin=[topics to join - separate by comma]&companyname=[company name]&contactname=[contact name]&contactemail=[contact email]&description=[topic description]&location=[location]&replicationfactor=[replication factor]&numpartitions=[number of partitions]&vipertoken=[viper token]
RETURNS: JSON object
getmaadsalgos:
http://[viperhost]:[viperport]/getmaadsalgos?maadsalgoserver=[MAADS server]&maadsport=[MAADS port]&maadsmicroservice=[MAADS microservice]&vipertoken=[VIPER token]&maadstoken=[MAADS server token]
RETURNS: JSON object
