Navigation

QUARTZ Tutorial

QUARTZ lets you schedule jobs at any time with ease.

1. Simple Repetitive Task

In this first example, we’ll use Quartz to repeat a simple task.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
%% copy the following into a file called: "repeat1.erl"
-module(repeat1).

-export([
         start/0
        ]).

start() ->
   %% start the default quartz server
   {ok, _} = quartz:start_link(),

   %% print "Hello Quartz (3sec.)" each 3sec.
   {ok, _} =quartz:apply_interval(3000, io, format, ["Hello Quartz (3sec.)~n"]),

   %% do the same, but each 10sec. (now with a "fun")
   {ok, _} = quartz:apply_interval(10000, fun() -> io:format("Hello Quartz (10sec.)~n") end).
Compile:
c:\> erlc repeat1.erl
Then run:
c:\> werl
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:2:2] [async-threads:0] [kernel-poll:false]
Eshell V5.9.1  (abort with ^G)

1> repeat1:start().

%% wait few seconds to see some outputs like these:
Hello Quartz (3sec.)
Hello Quartz (3sec.)
Hello Quartz (3sec.)
Hello Quartz (10sec.)

2> q().

Recap

  • Lines 2-6: declare a new module called repeat1 which exports a public function called start/0.
  • Line 10: start a new Quartz’s server.
  • Line 13: start a new timer that executes the {M,F,A} each 3 seconds ({M,F,A} equals {io, format, [“Hello Quartz (3sec.)~n”]}).
  • Line 16: start a new timer that executes a fun each 10 seconds.



2. Basic Usage

Below, we’ll describe some primitives to get started with Quartz.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
%% copy the following into a file called: "repeat2.erl"
-module(repeat2).

-export([
         start/0
        ]).

start() ->
   %% ------------------------------
   %% start the default quartz server
   {ok, QPid} = quartz:start_link(),

   %% print "Hello Word" once after 10sec.
   {ok, _} = quartz:apply_after(10000, fun() -> io:format("Hello World~n") end),

   %% repeatedly print "Hello WebArchiving" each 3sec.
   {ok, TRef1} = quartz:apply_interval(3000, io, format, ["Hello WebArchiving~n"]),

   %% ------------------------------
   %% start a new quartz server using namespace 'ns'
   {ok, NSPid} = quartz:start_link(ns),

   %% repeatedly print "Hello Erlang" each 5sec. using quartz server 'ns'
   {ok, _} = quartz:apply_interval(ns, 5000, io, format, ["Hello WebArchiving~n"]),

   %% [...] DO SOME WORK HERE

   %% cancel the timer referenced by TRef1
   ok = quartz:cancel(TRef1),

   %% cancel all timers in namespace 'ns'
   ok = quartz:cancel_all(ns),

   %% stop the default quartz server
   ok = quartz:stop(QPid), %% equivalent to quartz:stop/0

   %% stop the quartz server under namespace 'ns'
   ok = quartz:stop('ns'). %% you can also use quartz:stop(NSPid)

Recap

  • Line 11: start a new Quartz’s server and we keep its Pid for a further usage.

  • Line 14: schedule a task (here a message display) after 10 seconds.

  • Line 17: schedule a task execution every 3 seconds, and we keep the timer reference for a further usage.
  • Line 21: start another Quartz’s server with a name space ‘ns’.
  • Line 24: repeat a message display every 5 seconds on the named Quartz’s server.
  • Line 29: cancel the first Quartz’s timer using its reference.
  • Line 32: cancel all the timers started in the ‘ns’ Quartz’s server.
  • Line 35: stop the first Quartz’s server using its Pid.
  • Line 38: stop the named Quartz’s server using its name space (‘ns’).



3. Unix “AT” Clone

The code below shows how easy one can mimic the Unix command AT.
Open a new Erlang shell and try these:
c:\> werl
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:2:2] [async-threads:0] [kernel-poll:false]
Eshell V5.9.1  (abort with ^G)

%% execute the fun() today at 12:49 (notice the double {{ }})
1> quartz:apply_at({{12,49,0}}, fun() -> io:format("Hello Quartz~n") end).

%% execute the fun() on July 3rd, 2048 at 8:32:45
2> quartz:apply_at({{2048,7,3}, {8,32,45}}, fun() -> io:format("Hello Quartz~n") end).

3> q().



4. Advanced Jobs-Scheduling

The third example shows you how to use Quartz to schedule complex timing tasks.

Difference with CRON

Unlike CRON, you can schedule a job with a UNION of time
specifications rather than duplicating the same job with multiple specifications.
This has the good side effect of not launching the job more than
once if your time specifications overlap.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
%% copy the following into a file called: "repeat3.erl"
-module(repeat3).

-export([
         start/0, start/3
        ]).

start() ->
    start('my_super_crawler', 'archive', ["https://www.facebook.com/"]).

start(M,F,A) ->
    %% start the default quartz server
    {ok, _} = quartz:start_link(),

    TimeSpecs = [
       %% at midnight of each last day of the month
       {{'*','*',last}, {0,0,0}},

       %% at midnight of each 31st day of the
       %% month (will skip Feb, Apr, etc.)
       {{'*','*',31}, {0,0,0}},

       %% weekly, each Saturday at noon
       {{'*','*','*'}, sat, {12,0,0}},

       %% every 1st, 7th and 28th at midnight
       {{'*', '*', [1, 7, 28]}, {0,0,0}}

       %% the first Saturday of each month at noon
       {{'*','*',{1,7}}, sat, {12,0,0}},

       %% each Sunday every 8 hours
       {{'*', '*', '*'}, sun, {{0,23, 8}, 0,0}}

       %% bi-monthly, Saturday every two weeks at 3pm
       {{'*','*',[{1,7},{15,21}]}, sat, {15,0,0}},

       %% handy shortcuts for TimeSpecs (see doc.)
       {{12, 0, 0}},
       {sat,     {16, 0, 0}},
       {monthly, {5, 0, 0}},

       %% dealing with "Daylight Saving Time" can be tricky.
       %% Assuming you're in UK, and want to run a job at
       %% 12:00 UTC every day, the following UNION is useful:
       [
        {{'*',  {11,2},        '*'}, {12,0,0}},
        {{'*',       3,      {1,9}}, {12,0,0}},
        {{'*',       3,  {10,last}}, {13,0,0}},
        {{'*',   {4,9},        '*'}, {13,0,0}},
        {{'*',      10,     {1,26}}, {13,0,0}},
        {{'*',      10,  {27,last}}, {12,0,0}}
       ]
    ],


   %% schedule the M,F,A using the "TimeSpecs" list
   lists:foreach(fun(Spec) ->
      {ok,_} = quartz:schedule(Spec, M, F, A).
   end, TimeSpecs).
This is self-explanatory.



5. Sampling Time

When dealing with Quartz‘s TimeSpecs, you may find yourself in a situation where you want
schedule a task in a far future or short periods (very close to each other).

Linked “Quartz”

Suppose you want to schedule some tasks by a process P, and if
it dies, unschedule all its tasks.
Easy! Create a timer in linked mode: quart:start_link(Name,’true’). By specifying ‘true’ as a second argument to start_link/2, Quartz will automatically link to the calling process. If the latter dies, all its corresponding tasks will be unscheduled and no more executed.
The call to quartz_sched:sample/2 will help you understand when Quartz will schedule your task.
Open a new Erlang shell and try these:
c:\> werl
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:2:2] [async-threads:0] [kernel-poll:false]
Eshell V5.9.1  (abort with ^G)

%% generate 1 sample
1> quartz_sched:sample({{'*','*',last}, {0,0,0}}, 1).
[{{2013,2,28},{0,0,0}}]

%% generate 10 samples
2>  quartz_sched:sample({{'*','*',{1,5}}, {0,0,0}}, 10).
[{{2013,3,1},{0,0,0}},
 {{2013,3,2},{0,0,0}},
 {{2013,3,3},{0,0,0}},
 {{2013,3,4},{0,0,0}},
 {{2013,3,5},{0,0,0}},
 {{2013,4,1},{0,0,0}},
 {{2013,4,2},{0,0,0}},
 {{2013,4,3},{0,0,0}},
 {{2013,4,4},{0,0,0}},
 {{2013,4,5},{0,0,0}}]

%% generate 3 samples
3> quartz_sched:sample({{'*','*',31}, {0,0,0}}, 3).
[{{2013,3,31},{0,0,0}},
 {{2013,5,31},{0,0,0}},
 {{2013,7,31},{0,0,0}}]

%% generate 3 samples
4> quartz_sched:sample({{'*','*',28}, {0,0,0}}, 3).
[{{2013,2,28},{0,0,0}},
{{2013,3,28},{0,0,0}},
{{2013,4,28},{0,0,0}}]

%% generate 10 samples
5> quartz_sched:sample({{'*',2,29}, '*', {0,0,0}}, 10).
[{{2016,2,29},{0,0,0}},
 {{2020,2,29},{0,0,0}},
 {{2024,2,29},{0,0,0}},
 {{2028,2,29},{0,0,0}},
 {{2032,2,29},{0,0,0}},
 {{2036,2,29},{0,0,0}},
 {{2040,2,29},{0,0,0}},
 {{2044,2,29},{0,0,0}},
 {{2048,2,29},{0,0,0}},
 {{2052,2,29},{0,0,0}}]

%% generate 10 samples
6> quartz_sched:sample({{'*',2,last}, '*', {0,0,0}}, 10).
[{{2013,2,28},{0,0,0}},
 {{2014,2,28},{0,0,0}},
 {{2015,2,28},{0,0,0}},
 {{2016,2,29},{0,0,0}},
 {{2017,2,28},{0,0,0}},
 {{2018,2,28},{0,0,0}},
 {{2019,2,28},{0,0,0}},
 {{2020,2,29},{0,0,0}},
 {{2021,2,28},{0,0,0}},
 {{2022,2,28},{0,0,0}}]

%% generate 15 samples
7> quartz_sched:sample({{'*','*','*'}, {0,{0,59,5},0}}, 15).
[{{2013,2,18},{0,0,0}},
 {{2013,2,18},{0,5,0}},
 {{2013,2,18},{0,10,0}},
 {{2013,2,18},{0,15,0}},
 {{2013,2,18},{0,20,0}},
 {{2013,2,18},{0,25,0}},
 {{2013,2,18},{0,30,0}},
 {{2013,2,18},{0,35,0}},
 {{2013,2,18},{0,40,0}},
 {{2013,2,18},{0,45,0}},
 {{2013,2,18},{0,50,0}},
 {{2013,2,18},{0,55,0}},
 {{2013,2,19},{0,0,0}},
 {{2013,2,19},{0,5,0}},
 {{2013,2,19},{0,10,0}}]

%% generate 10 samples
8> quartz_sched:sample({{'*','*','*'}, {{0,23,3},0,0}}, 10).
[{{2013,2,17},{15,0,0}},
 {{2013,2,17},{18,0,0}},
 {{2013,2,17},{21,0,0}},
 {{2013,2,18},{0,0,0}},
 {{2013,2,18},{3,0,0}},
 {{2013,2,18},{6,0,0}},
 {{2013,2,18},{9,0,0}},
 {{2013,2,18},{12,0,0}},
 {{2013,2,18},{15,0,0}},
 {{2013,2,18},{18,0,0}}]

%% generate 5 samples
9> quartz_sched:sample({{'*','*','*'}, wed, {0,0,0}}, 5).
[{{2013,2,20},{0,0,0}},
 {{2013,2,27},{0,0,0}},
 {{2013,3,6},{0,0,0}},
 {{2013,3,13},{0,0,0}},
 {{2013,3,20},{0,0,0}}]

%% generate 5 samples
10> quartz_sched:sample({{'*','*','*'}, [sat,wed], {0,0,0}}, 5).
[{{2013,2,20},{0,0,0}},
 {{2013,2,23},{0,0,0}},
 {{2013,2,27},{0,0,0}},
 {{2013,3,2},{0,0,0}},
 {{2013,3,6},{0,0,0}}]

%% generate 10 samples
11> quartz_sched:sample({{'*','*',[{1,7},{15,21}]}, sun, {15,0,0}}, 10).
[{{2013,2,17},{15,0,0}},
 {{2013,3,3},{15,0,0}},
 {{2013,3,17},{15,0,0}},
 {{2013,4,7},{15,0,0}},
 {{2013,4,21},{15,0,0}},
 {{2013,5,5},{15,0,0}},
 {{2013,5,19},{15,0,0}},
 {{2013,6,2},{15,0,0}},
 {{2013,6,16},{15,0,0}},
 {{2013,7,7},{15,0,0}}]

%% generate 10 samples
12> quartz_sched:sample({{'*','*','*'}, {fri,sun}, {15,0,0}}, 10).
[{{2013,2,17},{15,0,0}},
 {{2013,2,22},{15,0,0}},
 {{2013,2,23},{15,0,0}},
 {{2013,2,24},{15,0,0}},
 {{2013,3,1},{15,0,0}},
 {{2013,3,2},{15,0,0}},
 {{2013,3,3},{15,0,0}},
 {{2013,3,8},{15,0,0}},
 {{2013,3,9},{15,0,0}},
 {{2013,3,10},{15,0,0}}]

%% generate 3 samples
13> quartz_sched:sample({{[2013,2014],12,31},{0,0,0}}, 3).
[{{2013,12,31},{0,0,0}},
 {{2014,12,31},{0,0,0}},
  error_unforeseeable_future]

%% generate 10 samples (AVOID TO TRAVEL THESE DAYS)
14> quartz_sched:sample({{'*','*',13}, fri, {0,0,0}},10).
[{{2013,9,13},{0,0,0}},
 {{2013,12,13},{0,0,0}},
 {{2014,6,13},{0,0,0}},
 {{2015,2,13},{0,0,0}},
 {{2015,3,13},{0,0,0}},
 {{2015,11,13},{0,0,0}},
 {{2016,5,13},{0,0,0}},
 {{2017,1,13},{0,0,0}},
 {{2017,10,13},{0,0,0}},
 {{2018,4,13},{0,0,0}}]

15> q().

6. Live Video Stream Archiving (RELOADED)

In this tutorial, we’ll enhance the Valeo example by adding support for scheduling using Quartz.
The original tutorial was about driving the VLC media player to capture a TV RTSP stream and archive it on disk.
Here, the live stream archiving task is scheduled and launched at precise dates allowing us to capture exactly the
TV show we’re interested in (the Hard Talk on BBC).
The following illustrates how scheduling can be easily achieved and quickly integrated to an existing application.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
%% overwrite the start/0 in the "vlc.erl" example (see VALEO) with the ones below.

[..]
-export([start/0, start/4]).

%%--------------------------------------------------------------------
%% @doc Drive VLC player command.
%% @end
%%--------------------------------------------------------------------
start() ->
  %% we're interested in a BBC World News Show called "Hard Talk" that
  %% airs each Tuesday, Wednesday and Thursday at 4:30 AM.
  Stream = "rtsp://media6.lsops.net:1935/live/bbcworld1_en_high.sdp",
  Output = "bbcwn-hard-talk-~p-~p-~p.mp4", %% output file format string
  Duration = 31*60, %% this is a 30mn program (+1mn for safety).
  Muxer = "ts",

  Schedule = {{*,*,*}, [wed, thu, tue], {4,30,0}} % each Tuesday, Wednesday, Thursday at 4:30 AM

  %% start the default quartz server
  {ok, _} = quartz:start_link(),

  %% this is where the magic happens
  {ok, _} = quartz:schedule(Schedule, ?MODULE, start, [Stream, Output, Duration, Muxer]).

start(Stream, Output, Duration, Muxer) ->
  {{Y,M,D},_} = calendar:now_to_locale_time(now()),
  OutputFileName = lists:flatten(io_lib:format(Output, [Y,M,D])),
  ProgArgs = [
              {cmd_name, find_vlc()},
              {cmd_params,
               ["-I dummy", %% run without GUI (console mode)
                "-q", %% quiet mode
                ?VLC_PARAMS(integer_to_list(Duration), Stream, OutputFileName, Muxer)
              ]}
             ],
  [..]
  %% The rest of the code is pretty much the same as in the original tutorial.
  %% This shows how easy to add "scheduling" to an existing code.


Combine Quartz and Valeo

Both modules can be combined to build robust, long-running applications.
At Aleph Archives, this combination is used to archive “in repetitive manner” our customer websites for years.

Recap

  • Line 24: Quartz will repeatedly call the vlc:start/4 function to grab the stream at the specified time slots (defined by time specification above: Tuesday, Wednesday, Thursday at 4:30 AM).



7. Archiving Twitter

In this last tutorial, we’ll use Quartz to follow a list of Twitter accounts.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
%% copy the following into a file called: "twitter_simple.erl"
-module(twitter_simple).

-export([start/0, start/1, fetch/2]).

%% the Twitter API request for an account and a number of results
-define(URL(Account,Count),
 "https://api.twitter.com/1/statuses/user_timeline.json?include_entities=true&include_rts=true&screen_name=" ++
 Account ++ "&count=" ++ integer_to_list(Count)).

-define(REFRESH, 60000).  %% default refresh time (60000ms = 1min.)
-define(COUNT, 10).  %% default number of (last) tweets to fetch

%% ------------------------------------------------------------
%% this module allows to follow a list of Twitter accounts with
%% a limited number of results per account
%% ------------------------------------------------------------
start() ->
  %% follow the default Twitter account with a default number of tweets to grab
  start([{"netpreserve", ?COUNT}]).

start([{Account, Count} | L]) ->
  %% start fetching the Twitter account every minute
  {ok, _} = quartz:apply_interval(?REFRESH, ?MODULE, fetch, [Account, Count]),

  %% continue with the remaining accounts, recurse!
  start(L);

start([]) ->
  ok.

fetch(Account, Count) ->
  %% Execute the fetching request on Twitter.
  %% For the sake of simplicity, there's no error handling.
  {ok, {{"HTTP/1.1",200,"OK"}, Hdrs, Tweets}} =
       httpc:request(get, {?URL(Account, Count), []}, [], []),

  io:format("Account:~s => ~s~n~n", [Account, Tweets]). %% print out the Tweets
Compile:
c:\> erlc twitter_simple.erl

The example uses the httpc module to send requests to Twitter. This requires to start some Erlang services before: ssl and inets.

Run:
c:\> werl -s ssl -s inets
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:2:2] [async-threads:0] [kernel-poll:false]
Eshell V5.9.1  (abort with ^G)

1> {ok, _} = quartz:start_link().

2> twitter_simple:start().

%% wait few seconds and watch the retrieved Tweets

3> q().

Recap

  • Lines 7-12: declare some marcos.
  • Line 20: call the start function with no parameters, this will execute a fetch on default account (i.e “netpreserve”).
  • Line 24: schedule a fetching task every 3 seconds with the call to quartz:apply_interval/4.
  • Line 27: repeat the schedule for other Twitter accounts.
  • Line 32: this fetching function fetch/2 will be repeatedly called by quartz:apply_interval/4 call.
  • Lines 35-36: use the httpc:request/4 call to send an HTTP GET request.
  • Lines 38: the Tweets are displayed (there’s no error handling to make the code short).

Note

A more interesting Twitter example is available at: twitter.erl
It can be used to archive not only Tweets, but also Tweets containing specific keywords.
Modify it to fits your needs.

Exercise

To keep this gem easy to read, we’re simply displaying the Tweets.
Use WSDK and save them in WARC files instead.
Hint: the variable Hdrs (line 35) contains the HTTP response headers.

Table Of Contents

API Reference

Navigation

© Copyright 2013, Aleph Archives. Created using Sphinx 1.1.3.