Database

Database connection

Mongo DB connection

class CveXplore.database.connection.mongo_db.MongoDBConnection(host='mongodb://127.0.0.1:27017', port=None, database='cvedb', **kwargs)

Bases: object

The MongoDBConnection class serves as a shell that functions as uniform way to connect to the mongodb backend. By default it will try to establish a connection towards a mongodb running on localhost (default port 27017) and database ‘cvedb’ (as per defaults of cve_search)

__del__()

Called when the class is garbage collected.

__init__(host='mongodb://127.0.0.1:27017', port=None, database='cvedb', **kwargs)

Parameters

• host (str) – The host parameter can be a full mongodb URI, in addition to a simple hostname.

• port (int) – Port number (optional when a URI is used as host parameter)

• database (str) – Database to connect to; defaults to cvedb (Cve Search default)

• kwargs (dict) – Other arguments supported by the MongoClient instantiation

__repr__()

String representation of object

disconnect()

Disconnect from mongodb

Database helpers

Generic database functions

class CveXplore.database.helpers.generic_db.GenericDatabaseFactory(collection)

Bases: DatasourceConnection

The GenericDatabaseFactory handles the creation of general, collection based, functions which provide an instance of CveXplore functions that apply to the given collection.

__init__(collection)

Create a new GenericDatabaseFactory and create field specific functions based on the __default_fields and the __fields_mapping.

Parameters

collection (str) – Collection to create the functions for

__repr__()

String representation of object

get_by_id(id)

Method to fetch a specific collection entry via it’s id number

Parameters

id (str) – Id number

Returns

Requested data or None

Return type

object

class CveXplore.database.helpers.generic_db.GenericDatabaseFieldsFunctions(field, collection)

Bases: DatasourceConnection

The GenericDatabaseFieldsFunctions handles the creation of general, field based, functions

__init__(field, collection)

Create a new GenericDatabaseFieldsFunctions and create field specific functions.

Parameters

• field (str) – Field name

• collection (str) – Collection to create the field functions for

__repr__()

String representation of object

find(value=None)

Method to find a given value.

Parameters

value (str) – Find a value

Returns

Data or None

Return type

object

search(value)

Method for searching for a given value. The value shall be converted to a regex.

Parameters

value (str) – Search for value

Returns

Data or None

Return type

object

Specific database functions

class CveXplore.database.helpers.specific_db.CvesDatabaseFunctions(collection)

Bases: GenericDatabaseFactory

The CvesDatabaseFunctions is a specific class that provides the cves attribute of a CveXplore instance additional functions that only apply to the ‘cves’ collection

__init__(collection)

Create a new GenericDatabaseFactory and create field specific functions based on the __default_fields and the __fields_mapping.

Parameters

collection (str) – Collection to create the functions for

__repr__()

String representation of object

get_cves_for_vendor(vendor, limit=0)

Function to return cves based on a given vendor. By default to result is sorted descending on th cvss field.

Parameters

• vendor (str) – A vendor to search for; e.g. microsoft

• limit (int) – Limit the amount of returned results

Returns

List with cves objects

Return type

list

Mongodb specific

class CveXplore.database.helpers.cvesearch_mongo_database.CveSearchCollection(database, name, **kwargs)

Bases: Collection

The CveSearchCollection is a custom Collection based on the pymongo Collection class which has been altered to return a CveSearchCursor reference on the find method.

__init__(database, name, **kwargs)

Get / create a custon cve-search Mongo collection.

Parameters

• database (MongoDBConnection) – the database to get a collection from

• name (str) – the name of the collection to get

• kwargs (kwargs) – additional keyword arguments will be passed as options for the create collection command

__repr__()

Return string representation of this class

find(*args, **kwargs)

Query the database as you would do so with a pymongo Collection.

Returns

Reference to the CveSearchCursor

Return type

CveSearchCursor

class CveXplore.database.helpers.cvesearch_mongo_database.CveSearchCursor(collection, *args, **kwargs)

Bases: Cursor

The CveSearchCursor is a custom cursor based on the pymongo cursor which will return database objects instead of the raw data from the mongodb database.

__init__(collection, *args, **kwargs)

Create a new cve-search cursor.

Parameters

collection (CveSearchCollection) – Reference to a CveSearchCollection object

__next__()

Advance the cursor and return CveXplore objects

__repr__()

Return string representation of this class

next()

Advance the cursor and return CveXplore objects

Database Maintenance

Main Updater

class CveXplore.database.maintenance.main_updater.MainUpdater(datasource)

Bases: object

The MainUpdater class is the main class for performing database maintenaince tasks

__init__(datasource)

Init a new MainUpdater class

Parameters

datasource (MongoDBConnection) – Datasource to update

initialize()

Method to initialize a new (fresh) instance of a cvedb database

update()

Method used for updating the database

Download Handler

class CveXplore.database.maintenance.DownloadHandler.DownloadHandler(feed_type, prefix=None)

Bases: ABC

DownloadHandler is the base class for all downloads and subsequent processing of the downloaded content. Each download script has a derived class which handles specifics for that type of content / download.

__init__(feed_type, prefix=None)

__repr__()

return string representation of object

chunk_list(lst, number)

Yield successive n-sized chunks from lst.

Parameters

• lst (list) – List to be chunked

• number (int) – Chunk size

Returns

Chunked list

Return type

list

get_session(retries=3, backoff_factor=0.3, status_forcelist=(429, 500, 502, 503, 504), session=None)

Method for returning a session object per every requesting thread

process_downloads(sites)

Method to download and process files

Parameters

sites (list) – List of file to download and process

Returns

Return type

store_file(response_content, content_type, url)

Method to store the download based on the headers content type

Parameters

• response_content (bytes) – Response content

• content_type (str) – Content type; e.g. ‘application/zip’

• url (str) – Download url

Returns

A working directory and a filename

Return type

str and str

File Handlers

class CveXplore.database.maintenance.file_handlers.JSONFileHandler(feed_type, prefix)

Bases: DownloadHandler

This class handles all JSON related download processing and functions as a base class for specific JSON sources processing and downloading

__init__(feed_type, prefix)

__repr__()

return string representation of object

file_to_queue(file_tuple)

Method responsible for transferring file contents to the worker queue for further processing and inserting them into the database

Parameters

file_tuple (tuple) – A tuple containing the working dir for processing and the filename of the file to process

class CveXplore.database.maintenance.file_handlers.XMLFileHandler(feed_type)

Bases: DownloadHandler

This class handles all XML related download processing and functions as a base class for specific XML sources processing and downloading

__init__(feed_type)

__repr__()

return string representation of object

process_item(item)

Method responsible for putting items into the worker queue as database actions

Parameters

item (dict) – Item processed from XML source and parsed into a dictionary

XML Content Handlers

class CveXplore.database.maintenance.content_handlers.CWEHandler

Bases: ContentHandler

Class handling the CWE XML parsing

__init__()

characters(ch)

Receive notification of character data.

The Parser will call this method to report each chunk of character data. SAX parsers may return all contiguous character data in a single chunk, or they may split it into several chunks; however, all of the characters in any single event must come from the same external entity so that the Locator provides useful information.

endElement(name)

Signals the end of an element in non-namespace mode.

The name parameter contains the name of the element type, just as with the startElement event.

startElement(name, attrs)

Signals the start of an element in non-namespace mode.

The name parameter contains the raw XML 1.0 name of the element type as a string and the attrs parameter holds an instance of the Attributes class containing the attributes of the element.

class CveXplore.database.maintenance.content_handlers.CapecHandler

Bases: ContentHandler

Class handling the CAPEC XML parsing

__init__()

characters(ch)

Receive notification of character data.

The Parser will call this method to report each chunk of character data. SAX parsers may return all contiguous character data in a single chunk, or they may split it into several chunks; however, all of the characters in any single event must come from the same external entity so that the Locator provides useful information.

endElement(name)

Signals the end of an element in non-namespace mode.

The name parameter contains the name of the element type, just as with the startElement event.

startElement(name, attrs)

Signals the start of an element in non-namespace mode.

The name parameter contains the raw XML 1.0 name of the element type as a string and the attrs parameter holds an instance of the Attributes class containing the attributes of the element.

Source processing classes

class CveXplore.database.maintenance.Sources_process.CAPECDownloads

Bases: XMLFileHandler

Class processing CAPEC source files

__init__()

class CveXplore.database.maintenance.Sources_process.CPEDownloads

Bases: JSONFileHandler

Class processing CPE source files

__init__()

class CveXplore.database.maintenance.Sources_process.CVEDownloads

Bases: JSONFileHandler

Class processing CVE source files

__init__()

static get_cve_year_range()

Method to fetch the years where we need cve’s for

class CveXplore.database.maintenance.Sources_process.CWEDownloads

Bases: XMLFileHandler

Class processing CWE source files

__init__()

class CveXplore.database.maintenance.Sources_process.DatabaseIndexer

Bases: object

Class processing the Mongodb indexes

__init__()

class CveXplore.database.maintenance.Sources_process.MongoAddIndex(index, name)

Bases: tuple

__getnewargs__()

Return self as a plain tuple. Used by copy and pickle.

static __new__(_cls, index, name)

Create new instance of MongoAddIndex(index, name)

__repr__()

Return a nicely formatted representation string

index

Alias for field number 0

name

Alias for field number 1

class CveXplore.database.maintenance.Sources_process.MongoUniqueIndex(index, name, unique)

Bases: tuple

__getnewargs__()

Return self as a plain tuple. Used by copy and pickle.

static __new__(_cls, index, name, unique)

Create new instance of MongoUniqueIndex(index, name, unique)

__repr__()

Return a nicely formatted representation string

index

Alias for field number 0

name

Alias for field number 1

unique

Alias for field number 2

class CveXplore.database.maintenance.Sources_process.VIADownloads

Bases: JSONFileHandler

Class processing VIA4 source files

__init__()

file_to_queue(file_tuple)

Method responsible for transferring file contents to the worker queue for further processing and inserting them into the database

Parameters

file_tuple (tuple) – A tuple containing the working dir for processing and the filename of the file to process

Log Handler

class CveXplore.database.maintenance.LogHandler.HelperLogger(name, level=0)

Bases: Logger

The HelperLogger is used by the application / gui as their logging class and extends the default python logger.logging class.

__init__(name, level=0)

Initialize the logger with a name and an optional level.

critical(msg, *args, **kwargs)

Log ‘msg % args’ with severity ‘CRITICAL’ and color RED.

Store logged message to the database for dashboard alerting.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

>>> logger.critical(“Houston, we have a hell of a problem”)

Parameters

msg (str) – Message to log

debug(msg, *args, **kwargs)

Log ‘msg % args’ with severity ‘DEBUG’ and color *MAGENTA.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

>>> logger.debug(“Houston, we have a thorny problem”)

Parameters

msg (str) – Message to log

error(msg, *args, **kwargs)

Log ‘msg % args’ with severity ‘ERROR’ and color RED.

Store logged message to the database for dashboard alerting.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

>>> logger.error(“Houston, we have a major problem”)

Parameters

msg (str) – Message to log

info(msg, *args, **kwargs)

Log ‘msg % args’ with severity ‘INFO’ and color WHITE.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

>>> logger.info(“Houston, we have an interesting problem”)

Parameters

msg (str) – Message to log

warning(msg, *args, **kwargs)

Log ‘msg % args’ with severity ‘WARNING’ and color YELLOW.

To pass exception information, use the keyword argument exc_info with a true value, e.g.

>>> logger.warning(“Houston, we have a bit of a problem”)

Parameters

msg (str) – Message to log

class CveXplore.database.maintenance.LogHandler.HostnameFilter(name='')

Bases: Filter

filter(record)

Determine if the specified record is to be logged.

Returns True if the record should be logged, or False otherwise. If deemed appropriate, the record may be modified in-place.

class CveXplore.database.maintenance.LogHandler.UpdateHandler(name, level=0)

Bases: HelperLogger

The UpdateHandler is used by the update process to provide written and visual feedback to the initiator of database management tasks.

__init__(name, level=0)

Initialize the logger with a name and an optional level.

Configuration

class CveXplore.database.maintenance.Config.Configuration

Bases: object

Class holding the configuration