Module wsdk_warc

Description
Function Index
Function Details

WARC factory module to access WARC files.

Version: 1.0.0

Authors: Aleph Archives Ltd. [web site: http://aleph-archives.com/].

Description

WARC factory module to access WARC files.

This is a pluggable interface that efficiently abstracts read/write/next operations to handle compressed (GZIP) or plain WARC files indifferently.

IMPORTANT NOTE

Other compression formats (Bzip2, Snappy, etc.) could be added easily, even though we, at Aleph Archives, do not recommend compression due to big performances drop during access.

Function Index

close/1	Dispose all resources in use by the current Iterator.
read/1	Opens a read-only WARC file Filename (compressed or not) at offset 0.
read/2	Opens a read-only WARC file Filename (compressed or not) at offset Offset.
write/1	Opens a WARC file Filename in write mode (compressed or not).
write/2	Writes the WARC record Record at the end of WARC file Warc.

Function Details

close/1

close(Warc::#wsdk_warc{}) -> ok

Dispose all resources in use by the current Iterator.

read/1

read(Filename::file:name() | #wsdk_warc{}) -> {ok, #wsdk_rrec{}, #wsdk_warc{}} | {invalid, non_neg_integer()} | eof

Equivalent to wsdk_read(Filename, 0).

Opens a read-only WARC file Filename (compressed or not) at offset 0.

Return an immediate working Iterator which can be used to iterate over WARC records using the same call wsdk_warc:read/1.

read/2

read(Filename::file:name(), Offset::non_neg_integer()) -> {ok, #wsdk_rrec{}, #wsdk_warc{}} | {invalid, non_neg_integer()} | eof

Opens a read-only WARC file Filename (compressed or not) at offset Offset.

Return an immediate working Iterator which can be used to iterate over WARC records using the same call wsdk_warc:read/1.

Note

The first call to this function takes Filename as an argument and return the tuple {ok, Record, Handle}. The returned handle is used in subsequent calls to iterate through the file's records.

So, a record is returned even during th first call. Remember, it's an iterator.

write/1

write(Filename::file:name()) -> {ok, #wsdk_warc{}}

Opens a WARC file Filename in write mode (compressed or not). The file will be opened for writing, and it will be created if it does not exist. Every write operation will take place at the end of the file.

write/2

write(Warc::#wsdk_warc{}, Record::#wsdk_wrec{}) -> {ok, #wsdk_warc{}}

Writes the WARC record Record at the end of WARC file Warc.

The operation is performed, if and only if, the record is syntactically and semantically valid (see wsdk_record:is_valid/1).

Generated by EDoc, Sep 5 2012, 17:38:09.