Merge branch 'master' of github.com:oasis-open/cti-python-stix2 + fix interoperability param support

.isort.cfg

@@ -4,7 +4,9 @@ not_skip = __init__.py
 known_third_party =
     antlr4,
     dateutil,
+    haversine,
     medallion,
+    pyjarowinkler,
     pytest,
     pytz,
     requests,

.travis.yml

@@ -1,16 +1,13 @@
 sudo: false
 language: python
 cache: pip
+dist: xenial
 python:
   - "2.7"
   - "3.4"
   - "3.5"
   - "3.6"
-matrix:
-  include:
-    - python: 3.7 # https://github.com/travis-ci/travis-ci/issues/9069#issuecomment-425720905
-      dist: xenial
-      sudo: true
+  - "3.7"
 install:
   - pip install -U pip setuptools
   - pip install tox-travis pre-commit

CHANGELOG

@@ -1,6 +1,15 @@
 CHANGELOG
 =========
 
+1.2.0 - 2019-09-25
+
+* #268, #271, #273, #275, #283, #285, #290 Changes support of STIX 2.1 to WD05 (CSD02), for all object types
+* #269 Updates id properties to take a spec_version parameter
+* #283 Changes the exception class hierarchy
+* #289 Adds functions for calculating semantic equivalence of two objects
+* #286 Fixes handling of custom observable extensions
+* #287 Fixes bug with timestamp precision preservation in MarkingDefinition objects
+
 1.1.3 - 2019-08-12
 
 * #258 Ignores empty values for optional fields

README.rst

@@ -1,4 +1,4 @@
-|Build_Status| |Coverage| |Version| |Downloads_Badge|
+|Build_Status| |Coverage| |Version| |Downloads_Badge| |Documentation_Status|
 
 cti-python-stix2
 ================
@@ -170,3 +170,6 @@ to repository-cla@oasis-open.org.
 .. |Downloads_Badge| image:: https://img.shields.io/pypi/dm/stix2.svg?maxAge=3600
    :target: https://pypi.python.org/pypi/stix2/
    :alt: Downloads
+.. |Documentation_Status| image:: https://readthedocs.org/projects/stix2/badge/?version=latest
+   :target: https://stix2.readthedocs.io/en/latest/?badge=latest
+   :alt: Documentation Status

docs/contributing.rst

@@ -109,3 +109,11 @@ then look at the resulting report in ``htmlcov/index.html``.
 All commits pushed to the ``master`` branch or submitted as a pull request are
 tested with `Travis-CI <https://travis-ci.org/oasis-open/cti-python-stix2>`_
 automatically.
+
+Adding a dependency
+-------------------
+
+One of the pre-commit hooks we use in our develoment environment enforces a
+consistent ordering to imports. If you need to add a new library as a dependency
+please add it to the `known_third_party` section of `.isort.cfg` to make sure
+the import is sorted correctly.

docs/guide/custom.ipynb

@@ -537,7 +537,7 @@
    "source": [
     "### Custom STIX Object Types\n",
     "\n",
-    "To create a custom STIX object type, define a class with the @[CustomObject](../api/stix2.v20.sdo.rst#stix2.v20.sdo.CustomObject) decorator. It takes the type name and a list of property tuples, each tuple consisting of the property name and a property instance. Any special validation of the properties can be added by supplying an ``__init__`` function.\n",
+    "To create a custom STIX object type, define a class with the @[CustomObject](../api/v20/stix2.v20.sdo.rst#stix2.v20.sdo.CustomObject) decorator. It takes the type name and a list of property tuples, each tuple consisting of the property name and a property instance. Any special validation of the properties can be added by supplying an ``__init__`` function.\n",
     "\n",
     "Let's say zoo animals have become a serious cyber threat and we want to model them in STIX using a custom object type. Let's use a ``species`` property to store the kind of animal, and make that property required. We also want a property to store the class of animal, such as \"mammal\" or \"bird\" but only want to allow specific values in it. We can add some logic to validate this property in ``__init__``."
    ]
@@ -841,7 +841,7 @@
    "source": [
     "### Custom Cyber Observable Types\n",
     "\n",
-    "Similar to custom STIX object types, use a decorator to create [custom Cyber Observable](../api/stix2.v20.observables.rst#stix2.v20.observables.CustomObservable) types. Just as before, ``__init__()`` can hold additional validation, but it is not necessary."
+    "Similar to custom STIX object types, use a decorator to create [custom Cyber Observable](../api/v20/stix2.v20.observables.rst#stix2.v20.observables.CustomObservable) types. Just as before, ``__init__()`` can hold additional validation, but it is not necessary."
    ]
   },
   {
@@ -1163,7 +1163,7 @@
    "source": [
     "### Custom Cyber Observable Extensions\n",
     "\n",
-    "Finally, custom extensions to existing Cyber Observable types can also be created. Just use the @[CustomExtension](../api/stix2.v20.observables.rst#stix2.v20.observables.CustomExtension) decorator. Note that you must provide the Cyber Observable class to which the extension applies. Again, any extra validation of the properties can be implemented by providing an ``__init__()`` but it is not required. Let's say we want to make an extension to the ``File`` Cyber Observable Object:"
+    "Finally, custom extensions to existing Cyber Observable types can also be created. Just use the @[CustomExtension](../api/v20/stix2.v20.observables.rst#stix2.v20.observables.CustomExtension) decorator. Note that you must provide the Cyber Observable class to which the extension applies. Again, any extra validation of the properties can be implemented by providing an ``__init__()`` but it is not required. Let's say we want to make an extension to the ``File`` Cyber Observable Object:"
    ]
   },
   {

docs/guide/datastore.ipynb

@@ -450,6 +450,14 @@
     "mem.source.filters.add([f1,f2])"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Note: The `defanged` property is now always included (implicitly) for STIX 2.1 Cyber Observable Objects (SCOs)**\n\n",
+    "This is important to remember if you are writing a filter that involves checking the `objects` property of a STIX 2.1 `ObservedData` object. If any of the objects associated with the `objects` property are STIX 2.1 SCOs, then your filter must include the `defanged` property. For an example, refer to `filters[14]` & `filters[15]` in stix2/test/v21/test_datastore_filters.py "
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -484,7 +492,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "If a STIX object has a `created_by_ref` property, you can use the [creator_of()](../api/stix2.datastore.rst#stix2.datastore.DataSource.creator_of) method to retrieve the [Identity](../api/stix2.v20.sdo.rst#stix2.v20.sdo.Identity) object that created it."
+    "If a STIX object has a `created_by_ref` property, you can use the [creator_of()](../api/stix2.datastore.rst#stix2.datastore.DataSource.creator_of) method to retrieve the [Identity](../api/v20/stix2.v20.sdo.rst#stix2.v20.sdo.Identity) object that created it."
    ]
   },
   {
@@ -726,21 +734,21 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "cti-python-stix2",
+   "display_name": "Python 3",
    "language": "python",
-   "name": "cti-python-stix2"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {
     "name": "ipython",
-    "version": 2
+    "version": 3
    },
    "file_extension": ".py",
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython2",
-   "version": "2.7.12"
+   "pygments_lexer": "ipython3",
+   "version": "3.6.7"
   }
  },
  "nbformat": 4,

docs/guide/environment.ipynb

@@ -67,7 +67,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 1,
    "metadata": {},
    "outputs": [],
    "source": [

docs/guide/ts_support.ipynb

@@ -365,7 +365,7 @@
    "source": [
     "### How custom content works\n",
     "\n",
-    "[CustomObject](../api/stix2.v20.sdo.rst#stix2.v20.sdo.CustomObject), [CustomObservable](../api/stix2.v20.observables.rst#stix2.v20.observables.CustomObservable), [CustomMarking](../api/stix2.v20.common.rst#stix2.v20.common.CustomMarking) and [CustomExtension](../api/stix2.v20.observables.rst#stix2.v20.observables.CustomExtension) must be registered explicitly by STIX version. This is a design decision since properties or requirements may change as the STIX Technical Specification advances.\n",
+    "[CustomObject](../api/v20/stix2.v20.sdo.rst#stix2.v20.sdo.CustomObject), [CustomObservable](../api/v20/stix2.v20.observables.rst#stix2.v20.observables.CustomObservable), [CustomMarking](../api/v20/stix2.v20.common.rst#stix2.v20.common.CustomMarking) and [CustomExtension](../api/v20/stix2.v20.observables.rst#stix2.v20.observables.CustomExtension) must be registered explicitly by STIX version. This is a design decision since properties or requirements may change as the STIX Technical Specification advances.\n",
     "\n",
     "You can perform this by:"
    ]

docs/guide/workbench.ipynb

@@ -624,7 +624,7 @@
    "source": [
     "### Creating STIX Data\n",
     "\n",
-    "To create a STIX object, just use that object's class constructor. Once it's created, add it to the workbench with [save()](../api/datastore/stix2.workbench.rst#stix2.workbench.save)."
+    "To create a STIX object, just use that object's class constructor. Once it's created, add it to the workbench with [save()](../api/stix2.workbench.rst#stix2.workbench.save)."
    ]
   },
   {
@@ -760,7 +760,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Defaults can also be set for the [created timestamp](../api/datastore/stix2.workbench.rst#stix2.workbench.set_default_created), [external references](../api/datastore/stix2.workbench.rst#stix2.workbench.set_default_external_refs) and [object marking references](../api/datastore/stix2.workbench.rst#stix2.workbench.set_default_object_marking_refs)."
+    "Defaults can also be set for the [created timestamp](../api/stix2.workbench.rst#stix2.workbench.set_default_created), [external references](../api/stix2.workbench.rst#stix2.workbench.set_default_external_refs) and [object marking references](../api/stix2.workbench.rst#stix2.workbench.set_default_object_marking_refs)."
    ]
   },
   {

requirements.txt

@@ -1,10 +1,10 @@
 bumpversion
 ipython
-nbsphinx==0.3.2
+nbsphinx==0.4.3
 pre-commit
 pytest
 pytest-cov
-sphinx<1.6
+sphinx<2
 sphinx-prompt
 tox
 

setup.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 1.1.3
+current_version = 1.2.0
 commit = True
 tag = True
 

setup.py

@@ -64,5 +64,6 @@ setup(
     },
     extras_require={
         'taxii': ['taxii2-client'],
+        'semantic': ['haversine', 'pyjarowinkler'],
     },
 )

stix2/__init__.py

@@ -20,6 +20,8 @@
 
 # flake8: noqa
 
+DEFAULT_VERSION = '2.0'  # Default version will always be the latest STIX 2.X version
+
 from .confidence import scales
 from .core import _collect_stix2_mappings, parse, parse_observable
 from .datastore import CompositeDataSource
@@ -56,5 +58,3 @@ from .v20 import *  # This import will always be the latest STIX 2.X version
 from .version import __version__
 
 _collect_stix2_mappings()
-
-DEFAULT_VERSION = '2.0'  # Default version will always be the latest STIX 2.X version

stix2/base.py

@@ -3,14 +3,17 @@
 import collections
 import copy
 import datetime as dt
+import uuid
 
 import simplejson as json
+import six
+
+from stix2.canonicalization.Canonicalize import canonicalize
 
 from .exceptions import (
-    AtLeastOnePropertyError, CustomContentError, DependentPropertiesError,
-    ExtraPropertiesError, ImmutableError, InvalidObjRefError,
-    InvalidValueError, MissingPropertiesError,
-    MutuallyExclusivePropertiesError,
+    AtLeastOnePropertyError, DependentPropertiesError, ExtraPropertiesError,
+    ImmutableError, InvalidObjRefError, InvalidValueError,
+    MissingPropertiesError, MutuallyExclusivePropertiesError,
 )
 from .markings.utils import validate
 from .utils import NOW, find_property_index, format_datetime, get_timestamp
@@ -20,6 +23,7 @@ from .utils import revoke as _revoke
 __all__ = ['STIXJSONEncoder', '_STIXBase']
 
 DEFAULT_ERROR = "{type} must have {property}='{expected}'."
+SCO_DET_ID_NAMESPACE = uuid.UUID("00abedb4-aa42-466c-9c01-fed23315a9b7")
 
 
 class STIXJSONEncoder(json.JSONEncoder):
@@ -88,10 +92,17 @@ class _STIXBase(collections.Mapping):
         if prop_name in kwargs:
             try:
                 kwargs[prop_name] = prop.clean(kwargs[prop_name])
-            except ValueError as exc:
-                if self.__allow_custom and isinstance(exc, CustomContentError):
-                    return
-                raise InvalidValueError(self.__class__, prop_name, reason=str(exc))
+            except InvalidValueError:
+                # No point in wrapping InvalidValueError in another
+                # InvalidValueError... so let those propagate.
+                raise
+            except Exception as exc:
+                six.raise_from(
+                    InvalidValueError(
+                        self.__class__, prop_name, reason=str(exc),
+                    ),
+                    exc,
+                )
 
     # interproperty constraint methods
 
@@ -105,10 +116,15 @@ class _STIXBase(collections.Mapping):
     def _check_at_least_one_property(self, list_of_properties=None):
         if not list_of_properties:
             list_of_properties = sorted(list(self.__class__._properties.keys()))
-            if 'type' in list_of_properties:
-                list_of_properties.remove('type')
+            if isinstance(self, _Observable):
+                props_to_remove = ["type", "id", "defanged", "spec_version"]
+            else:
+                props_to_remove = ["type"]
+
+            list_of_properties = [prop for prop in list_of_properties if prop not in props_to_remove]
         current_properties = self.properties_populated()
         list_of_properties_populated = set(list_of_properties).intersection(current_properties)
+
         if list_of_properties and (not list_of_properties_populated or list_of_properties_populated == set(['extensions'])):
             raise AtLeastOnePropertyError(self.__class__, list_of_properties)
 
@@ -295,9 +311,26 @@ class _Observable(_STIXBase):
         self.__allow_custom = kwargs.get('allow_custom', False)
         self._properties['extensions'].allow_custom = kwargs.get('allow_custom', False)
 
+        try:
+            # Since `spec_version` is optional, this is how we check for a 2.1 SCO
+            self._id_contributing_properties
+
+            if 'id' not in kwargs:
+                possible_id = self._generate_id(kwargs)
+                if possible_id is not None:
+                    kwargs['id'] = possible_id
+        except AttributeError:
+            # End up here if handling a 2.0 SCO, and don't need to do anything further
+            pass
+
         super(_Observable, self).__init__(**kwargs)
 
     def _check_ref(self, ref, prop, prop_name):
+        """
+        Only for checking `*_ref` or `*_refs` properties in spec_version 2.0
+        STIX Cyber Observables (SCOs)
+        """
+
         if '*' in self._STIXBase__valid_refs:
             return  # don't check if refs are valid
 
@@ -326,12 +359,51 @@ class _Observable(_STIXBase):
         if prop_name not in kwargs:
             return
 
+        from .properties import ObjectReferenceProperty
         if prop_name.endswith('_ref'):
-            ref = kwargs[prop_name]
-            self._check_ref(ref, prop, prop_name)
-        elif prop_name.endswith('_refs'):
-            for ref in kwargs[prop_name]:
+            if isinstance(prop, ObjectReferenceProperty):
+                ref = kwargs[prop_name]
                 self._check_ref(ref, prop, prop_name)
+        elif prop_name.endswith('_refs'):
+            if isinstance(prop.contained, ObjectReferenceProperty):
+                for ref in kwargs[prop_name]:
+                    self._check_ref(ref, prop, prop_name)
+
+    def _generate_id(self, kwargs):
+        required_prefix = self._type + "--"
+
+        properties_to_use = self._id_contributing_properties
+        if properties_to_use:
+            streamlined_obj_vals = []
+            if "hashes" in kwargs and "hashes" in properties_to_use:
+                possible_hash = _choose_one_hash(kwargs["hashes"])
+                if possible_hash:
+                    streamlined_obj_vals.append(possible_hash)
+            for key in properties_to_use:
+                if key != "hashes" and key in kwargs:
+                    if isinstance(kwargs[key], dict) or isinstance(kwargs[key], _STIXBase):
+                        temp_deep_copy = copy.deepcopy(dict(kwargs[key]))
+                        _recursive_stix_to_dict(temp_deep_copy)
+                        streamlined_obj_vals.append(temp_deep_copy)
+                    elif isinstance(kwargs[key], list) and isinstance(kwargs[key][0], _STIXBase):
+                        for obj in kwargs[key]:
+                            temp_deep_copy = copy.deepcopy(dict(obj))
+                            _recursive_stix_to_dict(temp_deep_copy)
+                            streamlined_obj_vals.append(temp_deep_copy)
+                    else:
+                        streamlined_obj_vals.append(kwargs[key])
+
+            if streamlined_obj_vals:
+                data = canonicalize(streamlined_obj_vals, utf8=False)
+
+                # try/except here to enable python 2 compatibility
+                try:
+                    return required_prefix + six.text_type(uuid.uuid5(SCO_DET_ID_NAMESPACE, data))
+                except UnicodeDecodeError:
+                    return required_prefix + six.text_type(uuid.uuid5(SCO_DET_ID_NAMESPACE, six.binary_type(data)))
+
+        # We return None if there are no values specified for any of the id-contributing-properties
+        return None
 
 
 class _Extension(_STIXBase):
@@ -341,6 +413,34 @@ class _Extension(_STIXBase):
         self._check_at_least_one_property()
 
 
+def _choose_one_hash(hash_dict):
+    if "MD5" in hash_dict:
+        return {"MD5": hash_dict["MD5"]}
+    elif "SHA-1" in hash_dict:
+        return {"SHA-1": hash_dict["SHA-1"]}
+    elif "SHA-256" in hash_dict:
+        return {"SHA-256": hash_dict["SHA-256"]}
+    elif "SHA-512" in hash_dict:
+        return {"SHA-512": hash_dict["SHA-512"]}
+    else:
+        k = next(iter(hash_dict), None)
+        if k is not None:
+            return {k: hash_dict[k]}
+
+
 def _cls_init(cls, obj, kwargs):
     if getattr(cls, '__init__', object.__init__) is not object.__init__:
         cls.__init__(obj, **kwargs)
+
+
+def _recursive_stix_to_dict(input_dict):
+    for key in input_dict:
+        if isinstance(input_dict[key], dict):
+            _recursive_stix_to_dict(input_dict[key])
+        elif isinstance(input_dict[key], _STIXBase):
+            input_dict[key] = dict(input_dict[key])
+
+            # There may stil be nested _STIXBase objects
+            _recursive_stix_to_dict(input_dict[key])
+        else:
+            return

stix2/canonicalization/Canonicalize.py

@@ -0,0 +1,512 @@
+##############################################################################
+#                                                                            #
+#  Copyright 2006-2019 WebPKI.org (http://webpki.org).                       #
+#                                                                            #
+#  Licensed under the Apache License, Version 2.0 (the "License");           #
+#  you may not use this file except in compliance with the License.          #
+#  You may obtain a copy of the License at                                   #
+#                                                                            #
+#      https://www.apache.org/licenses/LICENSE-2.0                           #
+#                                                                            #
+#  Unless required by applicable law or agreed to in writing, software       #
+#  distributed under the License is distributed on an "AS IS" BASIS,         #
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  #
+#  See the License for the specific language governing permissions and       #
+#  limitations under the License.                                            #
+#                                                                            #
+##############################################################################
+
+#################################################
+# JCS compatible JSON serializer for Python 3.x #
+#################################################
+
+# This file has been modified to be compatible with Python 2.x as well
+
+import re
+
+import six
+
+from stix2.canonicalization.NumberToJson import convert2Es6Format
+
+try:
+    from _json import encode_basestring_ascii as c_encode_basestring_ascii
+except ImportError:
+    c_encode_basestring_ascii = None
+try:
+    from _json import encode_basestring as c_encode_basestring
+except ImportError:
+    c_encode_basestring = None
+try:
+    from _json import make_encoder as c_make_encoder
+except ImportError:
+    c_make_encoder = None
+
+ESCAPE = re.compile(r'[\x00-\x1f\\"\b\f\n\r\t]')
+ESCAPE_ASCII = re.compile(r'([\\"]|[^\ -~])')
+HAS_UTF8 = re.compile(b'[\x80-\xff]')
+ESCAPE_DCT = {
+    '\\': '\\\\',
+    '"': '\\"',
+    '\b': '\\b',
+    '\f': '\\f',
+    '\n': '\\n',
+    '\r': '\\r',
+    '\t': '\\t',
+}
+for i in range(0x20):
+    ESCAPE_DCT.setdefault(chr(i), '\\u{0:04x}'.format(i))
+
+INFINITY = float('inf')
+
+
+def py_encode_basestring(s):
+    """Return a JSON representation of a Python string
+
+    """
+    def replace(match):
+        return ESCAPE_DCT[match.group(0)]
+    return '"' + ESCAPE.sub(replace, s) + '"'
+
+
+encode_basestring = (c_encode_basestring or py_encode_basestring)
+
+
+def py_encode_basestring_ascii(s):
+    """Return an ASCII-only JSON representation of a Python string
+
+    """
+    def replace(match):
+        s = match.group(0)
+        try:
+            return ESCAPE_DCT[s]
+        except KeyError:
+            n = ord(s)
+            if n < 0x10000:
+                return '\\u{0:04x}'.format(n)
+            else:
+                # surrogate pair
+                n -= 0x10000
+                s1 = 0xd800 | ((n >> 10) & 0x3ff)
+                s2 = 0xdc00 | (n & 0x3ff)
+                return '\\u{0:04x}\\u{1:04x}'.format(s1, s2)
+    return '"' + ESCAPE_ASCII.sub(replace, s) + '"'
+
+
+encode_basestring_ascii = (
+    c_encode_basestring_ascii or py_encode_basestring_ascii
+)
+
+
+class JSONEncoder(object):
+    """Extensible JSON <http://json.org> encoder for Python data structures.
+
+    Supports the following objects and types by default:
+
+    +-------------------+---------------+
+    | Python            | JSON          |
+    +===================+===============+
+    | dict              | object        |
+    +-------------------+---------------+
+    | list, tuple       | array         |
+    +-------------------+---------------+
+    | str               | string        |
+    +-------------------+---------------+
+    | int, float        | number        |
+    +-------------------+---------------+
+    | True              | true          |
+    +-------------------+---------------+
+    | False             | false         |
+    +-------------------+---------------+
+    | None              | null          |
+    +-------------------+---------------+
+
+    To extend this to recognize other objects, subclass and implement a
+    ``.default()`` method with another method that returns a serializable
+    object for ``o`` if possible, otherwise it should call the superclass
+    implementation (to raise ``TypeError``).
+
+    """
+    item_separator = ', '
+    key_separator = ': '
+
+    def __init__(
+        self, skipkeys=False, ensure_ascii=False,
+        check_circular=True, allow_nan=True, sort_keys=True,
+        indent=None, separators=(',', ':'), default=None,
+    ):
+        """Constructor for JSONEncoder, with sensible defaults.
+
+        If skipkeys is false, then it is a TypeError to attempt
+        encoding of keys that are not str, int, float or None.  If
+        skipkeys is True, such items are simply skipped.
+
+        If ensure_ascii is true, the output is guaranteed to be str
+        objects with all incoming non-ASCII characters escaped.  If
+        ensure_ascii is false, the output can contain non-ASCII characters.
+
+        If check_circular is true, then lists, dicts, and custom encoded
+        objects will be checked for circular references during encoding to
+        prevent an infinite recursion (which would cause an OverflowError).
+        Otherwise, no such check takes place.
+
+        If allow_nan is true, then NaN, Infinity, and -Infinity will be
+        encoded as such.  This behavior is not JSON specification compliant,
+        but is consistent with most JavaScript based encoders and decoders.
+        Otherwise, it will be a ValueError to encode such floats.
+
+        If sort_keys is true, then the output of dictionaries will be
+        sorted by key; this is useful for regression tests to ensure
+        that JSON serializations can be compared on a day-to-day basis.
+
+        If indent is a non-negative integer, then JSON array
+        elements and object members will be pretty-printed with that
+        indent level.  An indent level of 0 will only insert newlines.
+        None is the most compact representation.
+
+        If specified, separators should be an (item_separator, key_separator)
+        tuple.  The default is (', ', ': ') if *indent* is ``None`` and
+        (',', ': ') otherwise.  To get the most compact JSON representation,
+        you should specify (',', ':') to eliminate whitespace.
+
+        If specified, default is a function that gets called for objects
+        that can't otherwise be serialized.  It should return a JSON encodable
+        version of the object or raise a ``TypeError``.
+
+        """
+
+        self.skipkeys = skipkeys
+        self.ensure_ascii = ensure_ascii
+        self.check_circular = check_circular
+        self.allow_nan = allow_nan
+        self.sort_keys = sort_keys
+        self.indent = indent
+        if separators is not None:
+            self.item_separator, self.key_separator = separators
+        elif indent is not None:
+            self.item_separator = ','
+        if default is not None:
+            self.default = default
+
+    def default(self, o):
+        """Implement this method in a subclass such that it returns
+        a serializable object for ``o``, or calls the base implementation
+        (to raise a ``TypeError``).
+
+        For example, to support arbitrary iterators, you could
+        implement default like this::
+
+            def default(self, o):
+                try:
+                    iterable = iter(o)
+                except TypeError:
+                    pass
+                else:
+                    return list(iterable)
+                # Let the base class default method raise the TypeError
+                return JSONEncoder.default(self, o)
+
+        """
+        raise TypeError(
+            "Object of type '%s' is not JSON serializable" %
+            o.__class__.__name__,
+        )
+
+    def encode(self, o):
+        """Return a JSON string representation of a Python data structure.
+
+        >>> from json.encoder import JSONEncoder
+        >>> JSONEncoder().encode({"foo": ["bar", "baz"]})
+        '{"foo": ["bar", "baz"]}'
+
+        """
+        # This is for extremely simple cases and benchmarks.
+        if isinstance(o, str):
+            if self.ensure_ascii:
+                return encode_basestring_ascii(o)
+            else:
+                return encode_basestring(o)
+        # This doesn't pass the iterator directly to ''.join() because the
+        # exceptions aren't as detailed.  The list call should be roughly
+        # equivalent to the PySequence_Fast that ''.join() would do.
+        chunks = self.iterencode(o, _one_shot=False)
+        if not isinstance(chunks, (list, tuple)):
+            chunks = list(chunks)
+        return ''.join(chunks)
+
+    def iterencode(self, o, _one_shot=False):
+        """Encode the given object and yield each string
+        representation as available.
+
+        For example::
+
+            for chunk in JSONEncoder().iterencode(bigobject):
+                mysocket.write(chunk)
+
+        """
+        if self.check_circular:
+            markers = {}
+        else:
+            markers = None
+        if self.ensure_ascii:
+            _encoder = encode_basestring_ascii
+        else:
+            _encoder = encode_basestring
+
+        def floatstr(
+            o, allow_nan=self.allow_nan,
+            _repr=float.__repr__, _inf=INFINITY, _neginf=-INFINITY,
+        ):
+            # Check for specials.  Note that this type of test is processor
+            # and/or platform-specific, so do tests which don't depend on the
+            # internals.
+
+            if o != o:
+                text = 'NaN'
+            elif o == _inf:
+                text = 'Infinity'
+            elif o == _neginf:
+                text = '-Infinity'
+            else:
+                return _repr(o)
+
+            if not allow_nan:
+                raise ValueError(
+                    "Out of range float values are not JSON compliant: " +
+                    repr(o),
+                )
+
+            return text
+
+        if (
+            _one_shot and c_make_encoder is not None
+            and self.indent is None
+        ):
+            _iterencode = c_make_encoder(
+                markers, self.default, _encoder, self.indent,
+                self.key_separator, self.item_separator, self.sort_keys,
+                self.skipkeys, self.allow_nan,
+            )
+        else:
+            _iterencode = _make_iterencode(
+                markers, self.default, _encoder, self.indent, floatstr,
+                self.key_separator, self.item_separator, self.sort_keys,
+                self.skipkeys, _one_shot,
+            )
+        return _iterencode(o, 0)
+
+
+def _make_iterencode(
+    markers, _default, _encoder, _indent, _floatstr,
+        _key_separator, _item_separator, _sort_keys, _skipkeys, _one_shot,
+        # HACK: hand-optimized bytecode; turn globals into locals
+        ValueError=ValueError,
+        dict=dict,
+        float=float,
+        id=id,
+        int=int,
+        isinstance=isinstance,
+        list=list,
+        str=str,
+        tuple=tuple,
+        _intstr=int.__str__,
+):
+
+    if _indent is not None and not isinstance(_indent, str):
+        _indent = ' ' * _indent
+
+    def _iterencode_list(lst, _current_indent_level):
+        if not lst:
+            yield '[]'
+            return
+        if markers is not None:
+            markerid = id(lst)
+            if markerid in markers:
+                raise ValueError("Circular reference detected")
+            markers[markerid] = lst
+        buf = '['
+        if _indent is not None:
+            _current_indent_level += 1
+            newline_indent = '\n' + _indent * _current_indent_level
+            separator = _item_separator + newline_indent
+            buf += newline_indent
+        else:
+            newline_indent = None
+            separator = _item_separator
+        first = True
+        for value in lst:
+            if first:
+                first = False
+            else:
+                buf = separator
+            if isinstance(value, str):
+                yield buf + _encoder(value)
+            elif value is None:
+                yield buf + 'null'
+            elif value is True:
+                yield buf + 'true'
+            elif value is False:
+                yield buf + 'false'
+            elif isinstance(value, int):
+                # Subclasses of int/float may override __str__, but we still
+                # want to encode them as integers/floats in JSON. One example
+                # within the standard library is IntEnum.
+                yield buf + convert2Es6Format(value)
+            elif isinstance(value, float):
+                # see comment above for int
+                yield buf + convert2Es6Format(value)
+            else:
+                yield buf
+                if isinstance(value, (list, tuple)):
+                    chunks = _iterencode_list(value, _current_indent_level)
+                elif isinstance(value, dict):
+                    chunks = _iterencode_dict(value, _current_indent_level)
+                else:
+                    chunks = _iterencode(value, _current_indent_level)
+                # Below line commented-out for python2 compatibility
+                # yield from chunks
+                for chunk in chunks:
+                    yield chunk
+        if newline_indent is not None:
+            _current_indent_level -= 1
+            yield '\n' + _indent * _current_indent_level
+        yield ']'
+        if markers is not None:
+            del markers[markerid]
+
+    def _iterencode_dict(dct, _current_indent_level):
+        if not dct:
+            yield '{}'
+            return
+        if markers is not None:
+            markerid = id(dct)
+            if markerid in markers:
+                raise ValueError("Circular reference detected")
+            markers[markerid] = dct
+        yield '{'
+        if _indent is not None:
+            _current_indent_level += 1
+            newline_indent = '\n' + _indent * _current_indent_level
+            item_separator = _item_separator + newline_indent
+            yield newline_indent
+        else:
+            newline_indent = None
+            item_separator = _item_separator
+        first = True
+        if _sort_keys:
+            items = sorted(dct.items(), key=lambda kv: kv[0].encode('utf-16_be'))
+        else:
+            items = dct.items()
+        for key, value in items:
+            # Replaced isinstance(key, str) with below to enable simultaneous python 2 & 3 compatibility
+            if isinstance(key, six.string_types) or isinstance(key, six.binary_type):
+                pass
+            # JavaScript is weakly typed for these, so it makes sense to
+            # also allow them.  Many encoders seem to do something like this.
+            elif isinstance(key, float):
+                # see comment for int/float in _make_iterencode
+                key = convert2Es6Format(key)
+            elif key is True:
+                key = 'true'
+            elif key is False:
+                key = 'false'
+            elif key is None:
+                key = 'null'
+            elif isinstance(key, int):
+                # see comment for int/float in _make_iterencode
+                key = convert2Es6Format(key)
+            elif _skipkeys:
+                continue
+            else:
+                raise TypeError("key " + repr(key) + " is not a string")
+            if first:
+                first = False
+            else:
+                yield item_separator
+            yield _encoder(key)
+            yield _key_separator
+            if isinstance(value, str):
+                yield _encoder(value)
+            elif value is None:
+                yield 'null'
+            elif value is True:
+                yield 'true'
+            elif value is False:
+                yield 'false'
+            elif isinstance(value, int):
+                # see comment for int/float in _make_iterencode
+                yield convert2Es6Format(value)
+            elif isinstance(value, float):
+                # see comment for int/float in _make_iterencode
+                yield convert2Es6Format(value)
+            else:
+                if isinstance(value, (list, tuple)):
+                    chunks = _iterencode_list(value, _current_indent_level)
+                elif isinstance(value, dict):
+                    chunks = _iterencode_dict(value, _current_indent_level)
+                else:
+                    chunks = _iterencode(value, _current_indent_level)
+                # Below line commented-out for python2 compatibility
+                # yield from chunks
+                for chunk in chunks:
+                    yield chunk
+        if newline_indent is not None:
+            _current_indent_level -= 1
+            yield '\n' + _indent * _current_indent_level
+        yield '}'
+        if markers is not None:
+            del markers[markerid]
+
+    def _iterencode(o, _current_indent_level):
+        # Replaced isinstance(o, str) with below to enable simultaneous python 2 & 3 compatibility
+        if isinstance(o, six.string_types) or isinstance(o, six.binary_type):
+            yield _encoder(o)
+        elif o is None:
+            yield 'null'
+        elif o is True:
+            yield 'true'
+        elif o is False:
+            yield 'false'
+        elif isinstance(o, int):
+            # see comment for int/float in _make_iterencode
+            yield convert2Es6Format(o)
+        elif isinstance(o, float):
+            # see comment for int/float in _make_iterencode
+            yield convert2Es6Format(o)
+        elif isinstance(o, (list, tuple)):
+            # Below line commented-out for python2 compatibility
+            # yield from _iterencode_list(o, _current_indent_level)
+            for thing in _iterencode_list(o, _current_indent_level):
+                yield thing
+        elif isinstance(o, dict):
+            # Below line commented-out for python2 compatibility
+            # yield from _iterencode_dict(o, _current_indent_level)
+            for thing in _iterencode_dict(o, _current_indent_level):
+                yield thing
+        else:
+            if markers is not None:
+                markerid = id(o)
+                if markerid in markers:
+                    raise ValueError("Circular reference detected")
+                markers[markerid] = o
+            o = _default(o)
+            # Below line commented-out for python2 compatibility
+            # yield from _iterencode(o, _current_indent_level)
+            for thing in _iterencode(o, _current_indent_level):
+                yield thing
+            if markers is not None:
+                del markers[markerid]
+    return _iterencode
+
+
+def canonicalize(obj, utf8=True):
+    textVal = JSONEncoder(sort_keys=True).encode(obj)
+    if utf8:
+        return textVal.encode()
+    return textVal
+
+
+def serialize(obj, utf8=True):
+    textVal = JSONEncoder(sort_keys=False).encode(obj)
+    if utf8:
+        return textVal.encode()
+    return textVal

stix2/canonicalization/NumberToJson.py

@@ -0,0 +1,95 @@
+##############################################################################
+#                                                                            #
+#  Copyright 2006-2019 WebPKI.org (http://webpki.org).                       #
+#                                                                            #
+#  Licensed under the Apache License, Version 2.0 (the "License");           #
+#  you may not use this file except in compliance with the License.          #
+#  You may obtain a copy of the License at                                   #
+#                                                                            #
+#      https://www.apache.org/licenses/LICENSE-2.0                           #
+#                                                                            #
+#  Unless required by applicable law or agreed to in writing, software       #
+#  distributed under the License is distributed on an "AS IS" BASIS,         #
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  #
+#  See the License for the specific language governing permissions and       #
+#  limitations under the License.                                            #
+#                                                                            #
+##############################################################################
+
+
+##################################################################
+# Convert a Python double/float into an ES6/V8 compatible string #
+##################################################################
+def convert2Es6Format(value):
+    # Convert double/float to str using the native Python formatter
+    fvalue = float(value)
+
+    # Zero is a special case.  The following line takes "-0" case as well
+    if fvalue == 0:
+        return '0'
+
+    # The rest of the algorithm works on the textual representation only
+    pyDouble = str(fvalue)
+
+    # The following line catches the "inf" and "nan" values returned by str(fvalue)
+    if pyDouble.find('n') >= 0:
+        raise ValueError("Invalid JSON number: " + pyDouble)
+
+    # Save sign separately, it doesn't have any role in the algorithm
+    pySign = ''
+    if pyDouble.find('-') == 0:
+        pySign = '-'
+        pyDouble = pyDouble[1:]
+
+    # Now we should only have valid non-zero values
+    pyExpStr = ''
+    pyExpVal = 0
+    q = pyDouble.find('e')
+    if q > 0:
+        # Grab the exponent and remove it from the number
+        pyExpStr = pyDouble[q:]
+        if pyExpStr[2:3] == '0':
+            # Supress leading zero on exponents
+            pyExpStr = pyExpStr[:2] + pyExpStr[3:]
+        pyDouble = pyDouble[0:q]
+        pyExpVal = int(pyExpStr[1:])
+
+    # Split number in pyFirst + pyDot + pyLast
+    pyFirst = pyDouble
+    pyDot = ''
+    pyLast = ''
+    q = pyDouble.find('.')
+    if q > 0:
+        pyDot = '.'
+        pyFirst = pyDouble[:q]
+        pyLast = pyDouble[q + 1:]
+
+    # Now the string is split into: pySign + pyFirst + pyDot + pyLast + pyExpStr
+    if pyLast == '0':
+        # Always remove trailing .0
+        pyDot = ''
+        pyLast = ''
+
+    if pyExpVal > 0 and pyExpVal < 21:
+        # Integers are shown as is with up to 21 digits
+        pyFirst += pyLast
+        pyLast = ''
+        pyDot = ''
+        pyExpStr = ''
+        q = pyExpVal - len(pyFirst)
+        while q >= 0:
+            q -= 1
+            pyFirst += '0'
+    elif pyExpVal < 0 and pyExpVal > -7:
+        # Small numbers are shown as 0.etc with e-6 as lower limit
+        pyLast = pyFirst + pyLast
+        pyFirst = '0'
+        pyDot = '.'
+        pyExpStr = ''
+        q = pyExpVal
+        while q < -1:
+            q += 1
+            pyLast = '0' + pyLast
+
+    # The resulting sub-strings are concatenated
+    return pySign + pyFirst + pyDot + pyLast + pyExpStr

stix2/core.py

@@ -186,19 +186,6 @@ def parse_observable(data, _valid_refs=None, allow_custom=False, version=None):
         raise CustomContentError("Can't parse unknown observable type '%s'! For custom observables, "
                                  "use the CustomObservable decorator." % obj['type'])
 
-    EXT_MAP = STIX2_OBJ_MAPS[v]['observable-extensions']
-
-    if 'extensions' in obj and obj['type'] in EXT_MAP:
-        for name, ext in obj['extensions'].items():
-            try:
-                ext_class = EXT_MAP[obj['type']][name]
-            except KeyError:
-                if not allow_custom:
-                    raise CustomContentError("Can't parse unknown extension type '%s'"
-                                             "for observable type '%s'!" % (name, obj['type']))
-            else:  # extension was found
-                obj['extensions'][name] = ext_class(allow_custom=allow_custom, **obj['extensions'][name])
-
     return obj_class(allow_custom=allow_custom, **obj)
 
 

stix2/environment.py

@@ -1,9 +1,15 @@
 """Python STIX2 Environment API."""
 
 import copy
+import logging
+import time
 
 from .core import parse as _parse
 from .datastore import CompositeDataSource, DataStoreMixin
+from .exceptions import SemanticEquivalenceUnsupportedTypeError
+from .utils import STIXdatetime, parse_into_datetime
+
+logger = logging.getLogger(__name__)
 
 
 class ObjectFactory(object):
@@ -186,3 +192,448 @@ class Environment(DataStoreMixin):
             return self.get(creator_id)
         else:
             return None
+
+    @staticmethod
+    def semantically_equivalent(obj1, obj2, **weight_dict):
+        """This method is meant to verify if two objects of the same type are
+        semantically equivalent.
+
+        Args:
+            obj1: A stix2 object instance
+            obj2: A stix2 object instance
+            weight_dict: A dictionary that can be used to override settings
+                in the semantic equivalence process
+
+        Returns:
+            float: A number between 0.0 and 100.0 as a measurement of equivalence.
+
+        Warning:
+            Course of Action, Intrusion-Set, Observed-Data, Report are not supported
+            by this implementation. Indicator pattern check is also limited.
+
+        Note:
+            This implementation follows the Committee Note on semantic equivalence.
+            see `the Committee Note <link here>`__.
+
+        """
+        # default weights used for the semantic equivalence process
+        weights = {
+            "attack-pattern": {
+                "name": 30,
+                "external_references": 70,
+                "method": _attack_pattern_checks,
+            },
+            "campaign": {
+                "name": 60,
+                "aliases": 40,
+                "method": _campaign_checks,
+            },
+            "course-of-action": {
+                "method": _course_of_action_checks,
+            },
+            "identity": {
+                "name": 60,
+                "identity_class": 20,
+                "sectors": 20,
+                "method": _identity_checks,
+            },
+            "indicator": {
+                "indicator_types": 15,
+                "pattern": 80,
+                "valid_from": 5,
+                "tdelta": 1,  # One day interval
+                "method": _indicator_checks,
+            },
+            "intrusion-set": {
+                "method": _intrusion_set_checks,
+            },
+            "location": {
+                "longitude_latitude": 34,
+                "region": 33,
+                "country": 33,
+                "threshold": 1000.0,
+                "method": _location_checks,
+            },
+            "malware": {
+                "malware_types": 20,
+                "name": 80,
+                "method": _malware_checks,
+            },
+            "observed-data": {
+                "method": _observed_data_checks,
+            },
+            "report": {
+                "method": _report_checks,
+            },
+            "threat-actor": {
+                "name": 60,
+                "threat_actor_types": 20,
+                "aliases": 20,
+                "method": _threat_actor_checks,
+            },
+            "tool": {
+                "tool_types": 20,
+                "name": 80,
+                "method": _tool_checks,
+            },
+            "vulnerability": {
+                "name": 30,
+                "external_references": 70,
+                "method": _vulnerability_checks,
+            },
+            "_internal": {
+                "ignore_spec_version": False,
+            },
+        }
+
+        if weight_dict:
+            weights.update(weight_dict)
+
+        type1, type2 = obj1["type"], obj2["type"]
+        ignore_spec_version = weights["_internal"]["ignore_spec_version"]
+
+        if type1 != type2:
+            raise ValueError('The objects to compare must be of the same type!')
+
+        if ignore_spec_version is False and obj1.get("spec_version", "2.0") != obj2.get("spec_version", "2.0"):
+            raise ValueError('The objects to compare must be of the same spec version!')
+
+        method = weights[type1]["method"]
+        matching_score, sum_weights = method(obj1, obj2, **weights[type1])
+
+        if sum_weights <= 0:
+            return 0
+
+        equivalence_score = (matching_score / sum_weights) * 100.0
+        return equivalence_score
+
+
+def check_property_present(prop, obj1, obj2):
+    """Helper method checks if a property is present on both objects."""
+    if prop in obj1 and prop in obj2:
+        return True
+    return False
+
+
+def partial_timestamp_based(t1, t2, tdelta):
+    """Performs a timestamp-based matching via checking how close one timestamp is to another.
+
+    Args:
+        t1: A datetime string or STIXdatetime object.
+        t2: A datetime string or STIXdatetime object.
+        tdelta (float): A given time delta. This number is multiplied by 86400 (1 day) to
+            extend or shrink your time change tolerance.
+
+    Returns:
+        float: Number between 0.0 and 1.0 depending on match criteria.
+
+    """
+    if not isinstance(t1, STIXdatetime):
+        t1 = parse_into_datetime(t1)
+    if not isinstance(t2, STIXdatetime):
+        t2 = parse_into_datetime(t2)
+    t1, t2 = time.mktime(t1.timetuple()), time.mktime(t2.timetuple())
+    return 1 - min(abs(t1 - t2) / (86400 * tdelta), 1)
+
+
+def partial_list_based(l1, l2):
+    """Performs a partial list matching via finding the intersection between common values.
+
+    Args:
+        l1: A list of values.
+        l2: A list of values.
+
+    Returns:
+        float: 1.0 if the value matches exactly, 0.0 otherwise.
+
+    """
+    l1_set, l2_set = set(l1), set(l2)
+    return len(l1_set.intersection(l2_set)) / max(len(l1), len(l2))
+
+
+def exact_match(val1, val2):
+    """Performs an exact value match based on two values
+
+    Args:
+        val1: A value suitable for an equality test.
+        val2: A value suitable for an equality test.
+
+    Returns:
+        float: 1.0 if the value matches exactly, 0.0 otherwise.
+
+    """
+    if val1 == val2:
+        return 1.0
+    return 0.0
+
+
+def partial_string_based(str1, str2):
+    """Performs a partial string match using the Jaro-Winkler distance algorithm.
+
+    Args:
+        str1: A string value to check.
+        str2: A string value to check.
+
+    Returns:
+        float: Number between 0.0 and 1.0 depending on match criteria.
+
+    """
+    from pyjarowinkler import distance
+    return distance.get_jaro_distance(str1, str2)
+
+
+def custom_pattern_based(pattern1, pattern2):
+    """Performs a matching on Indicator Patterns.
+
+    Args:
+        pattern1: An Indicator pattern
+        pattern2: An Indicator pattern
+
+    Returns:
+        float: Number between 0.0 and 1.0 depending on match criteria.
+
+    """
+    logger.warning("Indicator pattern equivalence is not fully defined; will default to zero if not completely identical")
+    return exact_match(pattern1, pattern2)  # TODO: Implement pattern based equivalence
+
+
+def partial_external_reference_based(refs1, refs2):
+    """Performs a matching on External References.
+
+    Args:
+        refs1: A list of external references.
+        refs2: A list of external references.
+
+    Returns:
+        float: Number between 0.0 and 1.0 depending on matches.
+
+    """
+    allowed = set(("veris", "cve", "capec", "mitre-attack"))
+    matches = 0
+
+    if len(refs1) >= len(refs2):
+        l1 = refs1
+        l2 = refs2
+    else:
+        l1 = refs2
+        l2 = refs1
+
+    for ext_ref1 in l1:
+        for ext_ref2 in l2:
+            sn_match = False
+            ei_match = False
+            url_match = False
+            source_name = None
+
+            if check_property_present("source_name", ext_ref1, ext_ref2):
+                if ext_ref1["source_name"] == ext_ref2["source_name"]:
+                    source_name = ext_ref1["source_name"]
+                    sn_match = True
+            if check_property_present("external_id", ext_ref1, ext_ref2):
+                if ext_ref1["external_id"] == ext_ref2["external_id"]:
+                    ei_match = True
+            if check_property_present("url", ext_ref1, ext_ref2):
+                if ext_ref1["url"] == ext_ref2["url"]:
+                    url_match = True
+
+            # Special case: if source_name is a STIX defined name and either
+            # external_id or url match then its a perfect match and other entries
+            # can be ignored.
+            if sn_match and (ei_match or url_match) and source_name in allowed:
+                return 1.0
+
+            # Regular check. If the source_name (not STIX-defined) or external_id or
+            # url matches then we consider the entry a match.
+            if (sn_match or ei_match or url_match) and source_name not in allowed:
+                matches += 1
+
+    return matches / max(len(refs1), len(refs2))
+
+
+def partial_location_distance(lat1, long1, lat2, long2, threshold):
+    """Given two coordinates perform a matching based on its distance using the Haversine Formula.
+
+    Args:
+        lat1: Latitude value for first coordinate point.
+        lat2: Latitude value for second coordinate point.
+        long1: Longitude value for first coordinate point.
+        long2: Longitude value for second coordinate point.
+        threshold (float): A kilometer measurement for the threshold distance between these two points.
+
+    Returns:
+        float: Number between 0.0 and 1.0 depending on match.
+
+    """
+    from haversine import haversine, Unit
+    distance = haversine((lat1, long1), (lat2, long2), unit=Unit.KILOMETERS)
+    return 1 - (distance / threshold)
+
+
+def _attack_pattern_checks(obj1, obj2, **weights):
+    matching_score = 0.0
+    sum_weights = 0.0
+    if check_property_present("name", obj1, obj2):
+        w = weights["name"]
+        sum_weights += w
+        matching_score += w * partial_string_based(obj1["name"], obj2["name"])
+    if check_property_present("external_references", obj1, obj2):
+        w = weights["external_references"]
+        sum_weights += w
+        matching_score += (
+                w *
+                partial_external_reference_based(obj1["external_references"], obj2["external_references"])
+        )
+    return matching_score, sum_weights
+
+
+def _campaign_checks(obj1, obj2, **weights):
+    matching_score = 0.0
+    sum_weights = 0.0
+    if check_property_present("name", obj1, obj2):
+        w = weights["name"]
+        sum_weights += w
+        matching_score += w * partial_string_based(obj1["name"], obj2["name"])
+    if check_property_present("aliases", obj1, obj2):
+        w = weights["aliases"]
+        sum_weights += w
+        matching_score += w * partial_list_based(obj1["aliases"], obj2["aliases"])
+    return matching_score, sum_weights
+
+
+def _identity_checks(obj1, obj2, **weights):
+    matching_score = 0.0
+    sum_weights = 0.0
+    if check_property_present("name", obj1, obj2):
+        w = weights["name"]
+        sum_weights += w
+        matching_score += w * exact_match(obj1["name"], obj2["name"])
+    if check_property_present("identity_class", obj1, obj2):
+        w = weights["identity_class"]
+        sum_weights += w
+        matching_score += w * exact_match(obj1["identity_class"], obj2["identity_class"])
+    if check_property_present("sectors", obj1, obj2):
+        w = weights["sectors"]
+        sum_weights += w
+        matching_score += w * partial_list_based(obj1["sectors"], obj2["sectors"])
+    return matching_score, sum_weights
+
+
+def _indicator_checks(obj1, obj2, **weights):
+    matching_score = 0.0
+    sum_weights = 0.0
+    if check_property_present("indicator_types", obj1, obj2):
+        w = weights["indicator_types"]
+        sum_weights += w
+        matching_score += w * partial_list_based(obj1["indicator_types"], obj2["indicator_types"])
+    if check_property_present("pattern", obj1, obj2):
+        w = weights["pattern"]
+        sum_weights += w
+        matching_score += w * custom_pattern_based(obj1["pattern"], obj2["pattern"])
+    if check_property_present("valid_from", obj1, obj2):
+        w = weights["valid_from"]
+        sum_weights += w
+        matching_score += (
+                w *
+                partial_timestamp_based(obj1["valid_from"], obj2["valid_from"], weights["tdelta"])
+        )
+    return matching_score, sum_weights
+
+
+def _location_checks(obj1, obj2, **weights):
+    matching_score = 0.0
+    sum_weights = 0.0
+    if check_property_present("latitude", obj1, obj2) and check_property_present("longitude", obj1, obj2):
+        w = weights["longitude_latitude"]
+        sum_weights += w
+        matching_score += (
+                w *
+                partial_location_distance(obj1["latitude"], obj1["longitude"], obj2["latitude"], obj2["longitude"], weights["threshold"])
+        )
+    if check_property_present("region", obj1, obj2):
+        w = weights["region"]
+        sum_weights += w
+        matching_score += w * exact_match(obj1["region"], obj2["region"])
+    if check_property_present("country", obj1, obj2):
+        w = weights["country"]
+        sum_weights += w
+        matching_score += w * exact_match(obj1["country"], obj2["country"])
+    return matching_score, sum_weights
+
+
+def _malware_checks(obj1, obj2, **weights):
+    matching_score = 0.0
+    sum_weights = 0.0
+    if check_property_present("malware_types", obj1, obj2):
+        w = weights["malware_types"]
+        sum_weights += w
+        matching_score += w * partial_list_based(obj1["malware_types"], obj2["malware_types"])
+    if check_property_present("name", obj1, obj2):
+        w = weights["name"]
+        sum_weights += w
+        matching_score += w * partial_string_based(obj1["name"], obj2["name"])
+    return matching_score, sum_weights
+
+
+def _threat_actor_checks(obj1, obj2, **weights):
+    matching_score = 0.0
+    sum_weights = 0.0
+    if check_property_present("name", obj1, obj2):
+        w = weights["name"]
+        sum_weights += w
+        matching_score += w * partial_string_based(obj1["name"], obj2["name"])
+    if check_property_present("threat_actor_types", obj1, obj2):
+        w = weights["threat_actor_types"]
+        sum_weights += w
+        matching_score += w * partial_list_based(obj1["threat_actor_types"], obj2["threat_actor_types"])
+    if check_property_present("aliases", obj1, obj2):
+        w = weights["aliases"]
+        sum_weights += w
+        matching_score += w * partial_list_based(obj1["aliases"], obj2["aliases"])
+    return matching_score, sum_weights
+
+
+def _tool_checks(obj1, obj2, **weights):
+    matching_score = 0.0
+    sum_weights = 0.0
+    if check_property_present("tool_types", obj1, obj2):
+        w = weights["tool_types"]
+        sum_weights += w
+        matching_score += w * partial_list_based(obj1["tool_types"], obj2["tool_types"])
+    if check_property_present("name", obj1, obj2):
+        w = weights["name"]
+        sum_weights += w
+        matching_score += w * partial_string_based(obj1["name"], obj2["name"])
+    return matching_score, sum_weights
+
+
+def _vulnerability_checks(obj1, obj2, **weights):
+    matching_score = 0.0
+    sum_weights = 0.0
+    if check_property_present("name", obj1, obj2):
+        w = weights["name"]
+        sum_weights += w
+        matching_score += w * partial_string_based(obj1["name"], obj2["name"])
+    if check_property_present("external_references", obj1, obj2):
+        w = weights["external_references"]
+        sum_weights += w
+        matching_score += w * partial_external_reference_based(
+            obj1["external_references"],
+            obj2["external_references"],
+        )
+    return matching_score, sum_weights
+
+
+def _course_of_action_checks(obj1, obj2, **weights):
+    raise SemanticEquivalenceUnsupportedTypeError("course-of-action type has no semantic equivalence implementation!")
+
+
+def _intrusion_set_checks(obj1, obj2, **weights):
+    raise SemanticEquivalenceUnsupportedTypeError("intrusion-set type has no semantic equivalence implementation!")
+
+
+def _observed_data_checks(obj1, obj2, **weights):
+    raise SemanticEquivalenceUnsupportedTypeError("observed-data type has no semantic equivalence implementation!")
+
+
+def _report_checks(obj1, obj2, **weights):
+    raise SemanticEquivalenceUnsupportedTypeError("report type has no semantic equivalence implementation!")

stix2/exceptions.py

@@ -5,7 +5,15 @@ class STIXError(Exception):
     """Base class for errors generated in the stix2 library."""
 
 
-class InvalidValueError(STIXError, ValueError):
+class ObjectConfigurationError(STIXError):
+    """
+    Represents specification violations regarding the composition of STIX
+    objects.
+    """
+    pass
+
+
+class InvalidValueError(ObjectConfigurationError):
     """An invalid value was provided to a STIX object's ``__init__``."""
 
     def __init__(self, cls, prop_name, reason):
@@ -19,52 +27,89 @@ class InvalidValueError(STIXError, ValueError):
         return msg.format(self)
 
 
-class MissingPropertiesError(STIXError, ValueError):
+class PropertyPresenceError(ObjectConfigurationError):
+    """
+    Represents an invalid combination of properties on a STIX object.  This
+    class can be used directly when the object requirements are more
+    complicated and none of the more specific exception subclasses apply.
+    """
+    def __init__(self, message, cls):
+        super(PropertyPresenceError, self).__init__(message)
+        self.cls = cls
+
+
+class MissingPropertiesError(PropertyPresenceError):
     """Missing one or more required properties when constructing STIX object."""
 
     def __init__(self, cls, properties):
-        super(MissingPropertiesError, self).__init__()
-        self.cls = cls
-        self.properties = sorted(list(properties))
+        self.properties = sorted(properties)
 
-    def __str__(self):
-        msg = "No values for required properties for {0}: ({1})."
-        return msg.format(
-            self.cls.__name__,
+        msg = "No values for required properties for {0}: ({1}).".format(
+            cls.__name__,
             ", ".join(x for x in self.properties),
         )
 
+        super(MissingPropertiesError, self).__init__(msg, cls)
 
-class ExtraPropertiesError(STIXError, TypeError):
+
+class ExtraPropertiesError(PropertyPresenceError):
     """One or more extra properties were provided when constructing STIX object."""
 
     def __init__(self, cls, properties):
-        super(ExtraPropertiesError, self).__init__()
-        self.cls = cls
-        self.properties = sorted(list(properties))
+        self.properties = sorted(properties)
 
-    def __str__(self):
-        msg = "Unexpected properties for {0}: ({1})."
-        return msg.format(
-            self.cls.__name__,
+        msg = "Unexpected properties for {0}: ({1}).".format(
+            cls.__name__,
             ", ".join(x for x in self.properties),
         )
 
-
-class ImmutableError(STIXError, ValueError):
-    """Attempted to modify an object after creation."""
-
-    def __init__(self, cls, key):
-        super(ImmutableError, self).__init__()
-        self.cls = cls
-        self.key = key
-
-    def __str__(self):
-        msg = "Cannot modify '{0.key}' property in '{0.cls.__name__}' after creation."
-        return msg.format(self)
+        super(ExtraPropertiesError, self).__init__(msg, cls)
 
 
-class DictionaryKeyError(STIXError, ValueError):
+class MutuallyExclusivePropertiesError(PropertyPresenceError):
+    """Violating interproperty mutually exclusive constraint of a STIX object type."""
+
+    def __init__(self, cls, properties):
+        self.properties = sorted(properties)
+
+        msg = "The ({1}) properties for {0} are mutually exclusive.".format(
+            cls.__name__,
+            ", ".join(x for x in self.properties),
+        )
+
+        super(MutuallyExclusivePropertiesError, self).__init__(msg, cls)
+
+
+class DependentPropertiesError(PropertyPresenceError):
+    """Violating interproperty dependency constraint of a STIX object type."""
+
+    def __init__(self, cls, dependencies):
+        self.dependencies = dependencies
+
+        msg = "The property dependencies for {0}: ({1}) are not met.".format(
+            cls.__name__,
+            ", ".join(name for x in self.dependencies for name in x),
+        )
+
+        super(DependentPropertiesError, self).__init__(msg, cls)
+
+
+class AtLeastOnePropertyError(PropertyPresenceError):
+    """Violating a constraint of a STIX object type that at least one of the given properties must be populated."""
+
+    def __init__(self, cls, properties):
+        self.properties = sorted(properties)
+
+        msg = "At least one of the ({1}) properties for {0} must be " \
+              "populated.".format(
+                   cls.__name__,
+                   ", ".join(x for x in self.properties),
+              )
+
+        super(AtLeastOnePropertyError, self).__init__(msg, cls)
+
+
+class DictionaryKeyError(ObjectConfigurationError):
     """Dictionary key does not conform to the correct format."""
 
     def __init__(self, key, reason):
@@ -77,7 +122,7 @@ class DictionaryKeyError(STIXError, ValueError):
         return msg.format(self)
 
 
-class InvalidObjRefError(STIXError, ValueError):
+class InvalidObjRefError(ObjectConfigurationError):
     """A STIX Cyber Observable Object contains an invalid object reference."""
 
     def __init__(self, cls, prop_name, reason):
@@ -91,95 +136,7 @@ class InvalidObjRefError(STIXError, ValueError):
         return msg.format(self)
 
 
-class UnmodifiablePropertyError(STIXError, ValueError):
-    """Attempted to modify an unmodifiable property of object when creating a new version."""
-
-    def __init__(self, unchangable_properties):
-        super(UnmodifiablePropertyError, self).__init__()
-        self.unchangable_properties = unchangable_properties
-
-    def __str__(self):
-        msg = "These properties cannot be changed when making a new version: {0}."
-        return msg.format(", ".join(self.unchangable_properties))
-
-
-class MutuallyExclusivePropertiesError(STIXError, TypeError):
-    """Violating interproperty mutually exclusive constraint of a STIX object type."""
-
-    def __init__(self, cls, properties):
-        super(MutuallyExclusivePropertiesError, self).__init__()
-        self.cls = cls
-        self.properties = sorted(list(properties))
-
-    def __str__(self):
-        msg = "The ({1}) properties for {0} are mutually exclusive."
-        return msg.format(
-            self.cls.__name__,
-            ", ".join(x for x in self.properties),
-        )
-
-
-class DependentPropertiesError(STIXError, TypeError):
-    """Violating interproperty dependency constraint of a STIX object type."""
-
-    def __init__(self, cls, dependencies):
-        super(DependentPropertiesError, self).__init__()
-        self.cls = cls
-        self.dependencies = dependencies
-
-    def __str__(self):
-        msg = "The property dependencies for {0}: ({1}) are not met."
-        return msg.format(
-            self.cls.__name__,
-            ", ".join(name for x in self.dependencies for name in x),
-        )
-
-
-class AtLeastOnePropertyError(STIXError, TypeError):
-    """Violating a constraint of a STIX object type that at least one of the given properties must be populated."""
-
-    def __init__(self, cls, properties):
-        super(AtLeastOnePropertyError, self).__init__()
-        self.cls = cls
-        self.properties = sorted(list(properties))
-
-    def __str__(self):
-        msg = "At least one of the ({1}) properties for {0} must be populated."
-        return msg.format(
-            self.cls.__name__,
-            ", ".join(x for x in self.properties),
-        )
-
-
-class RevokeError(STIXError, ValueError):
-    """Attempted to an operation on a revoked object."""
-
-    def __init__(self, called_by):
-        super(RevokeError, self).__init__()
-        self.called_by = called_by
-
-    def __str__(self):
-        if self.called_by == "revoke":
-            return "Cannot revoke an already revoked object."
-        else:
-            return "Cannot create a new version of a revoked object."
-
-
-class ParseError(STIXError, ValueError):
-    """Could not parse object."""
-
-    def __init__(self, msg):
-        super(ParseError, self).__init__(msg)
-
-
-class CustomContentError(STIXError, ValueError):
-    """Custom STIX Content (SDO, Observable, Extension, etc.) detected."""
-
-    def __init__(self, msg):
-        super(CustomContentError, self).__init__(msg)
-
-
-class InvalidSelectorError(STIXError, AssertionError):
+class InvalidSelectorError(ObjectConfigurationError):
     """Granular Marking selector violation. The selector must resolve into an existing STIX object property."""
 
     def __init__(self, cls, key):
@@ -192,7 +149,73 @@ class InvalidSelectorError(STIXError, AssertionError):
         return msg.format(self.key, self.cls.__class__.__name__)
 
 
-class MarkingNotFoundError(STIXError, AssertionError):
+class TLPMarkingDefinitionError(ObjectConfigurationError):
+    """Marking violation. The marking-definition for TLP MUST follow the mandated instances from the spec."""
+
+    def __init__(self, user_obj, spec_obj):
+        super(TLPMarkingDefinitionError, self).__init__()
+        self.user_obj = user_obj
+        self.spec_obj = spec_obj
+
+    def __str__(self):
+        msg = "Marking {0} does not match spec marking {1}!"
+        return msg.format(self.user_obj, self.spec_obj)
+
+
+class ImmutableError(STIXError):
+    """Attempted to modify an object after creation."""
+
+    def __init__(self, cls, key):
+        super(ImmutableError, self).__init__()
+        self.cls = cls
+        self.key = key
+
+    def __str__(self):
+        msg = "Cannot modify '{0.key}' property in '{0.cls.__name__}' after creation."
+        return msg.format(self)
+
+
+class UnmodifiablePropertyError(STIXError):
+    """Attempted to modify an unmodifiable property of object when creating a new version."""
+
+    def __init__(self, unchangable_properties):
+        super(UnmodifiablePropertyError, self).__init__()
+        self.unchangable_properties = unchangable_properties
+
+    def __str__(self):
+        msg = "These properties cannot be changed when making a new version: {0}."
+        return msg.format(", ".join(self.unchangable_properties))
+
+
+class RevokeError(STIXError):
+    """Attempted an operation on a revoked object."""
+
+    def __init__(self, called_by):
+        super(RevokeError, self).__init__()
+        self.called_by = called_by
+
+    def __str__(self):
+        if self.called_by == "revoke":
+            return "Cannot revoke an already revoked object."
+        else:
+            return "Cannot create a new version of a revoked object."
+
+
+class ParseError(STIXError):
+    """Could not parse object."""
+
+    def __init__(self, msg):
+        super(ParseError, self).__init__(msg)
+
+
+class CustomContentError(STIXError):
+    """Custom STIX Content (SDO, Observable, Extension, etc.) detected."""
+
+    def __init__(self, msg):
+        super(CustomContentError, self).__init__(msg)
+
+
+class MarkingNotFoundError(STIXError):
     """Marking violation. The marking reference must be present in SDO or SRO."""
 
     def __init__(self, cls, key):
@@ -205,14 +228,15 @@ class MarkingNotFoundError(STIXError, AssertionError):
         return msg.format(self.key, self.cls.__class__.__name__)
 
 
-class TLPMarkingDefinitionError(STIXError, AssertionError):
-    """Marking violation. The marking-definition for TLP MUST follow the mandated instances from the spec."""
+class STIXDeprecationWarning(DeprecationWarning):
+    """
+    Represents usage of a deprecated component of a STIX specification.
+    """
+    pass
 
-    def __init__(self, user_obj, spec_obj):
-        super(TLPMarkingDefinitionError, self).__init__()
-        self.user_obj = user_obj
-        self.spec_obj = spec_obj
 
-    def __str__(self):
-        msg = "Marking {0} does not match spec marking {1}!"
-        return msg.format(self.user_obj, self.spec_obj)
+class SemanticEquivalenceUnsupportedTypeError(STIXError, TypeError):
+    """STIX object type not supported by the semantic equivalence approach."""
+
+    def __init__(self, msg):
+        super(SemanticEquivalenceUnsupportedTypeError, self).__init__(msg)

stix2/properties.py

@@ -11,36 +11,82 @@ import uuid
 from six import string_types, text_type
 from stix2patterns.validator import run_validator
 
+import stix2
+
 from .base import _Observable, _STIXBase
 from .core import STIX2_OBJ_MAPS, parse, parse_observable
-from .exceptions import CustomContentError, DictionaryKeyError
+from .exceptions import (
+    CustomContentError, DictionaryKeyError, MissingPropertiesError,
+    MutuallyExclusivePropertiesError,
+)
 from .utils import _get_dict, get_class_hierarchy_names, parse_into_datetime
 
-# This uses the regular expression for a RFC 4122, Version 4 UUID. In the
-# 8-4-4-4-12 hexadecimal representation, the first hex digit of the third
-# component must be a 4, and the first hex digit of the fourth component
-# must be 8, 9, a, or b (10xx bit pattern).
-ID_REGEX = re.compile(
-    r"^[a-z0-9][a-z0-9-]+[a-z0-9]--"  # object type
-    "[0-9a-fA-F]{8}-"
-    "[0-9a-fA-F]{4}-"
-    "4[0-9a-fA-F]{3}-"
-    "[89abAB][0-9a-fA-F]{3}-"
-    "[0-9a-fA-F]{12}$",
-)
-
-ID_REGEX_interoperability = re.compile(r"^[a-z0-9][a-z0-9-]+[a-z0-9]--"  # object type
-                                       "[0-9a-fA-F]{8}-"
+ID_REGEX_interoperability = re.compile(r"[0-9a-fA-F]{8}-"
                                        "[0-9a-fA-F]{4}-"
                                        "[0-9a-fA-F]{4}-"
                                        "[0-9a-fA-F]{4}-"
                                        "[0-9a-fA-F]{12}$")
 
 ERROR_INVALID_ID = (
-    "not a valid STIX identifier, must match <object-type>--<UUIDv4>"
+    "not a valid STIX identifier, must match <object-type>--<UUID>: {}"
 )
 
 
+def _check_uuid(uuid_str, spec_version, interoperability):
+    """
+    Check whether the given UUID string is valid with respect to the given STIX
+    spec version.  STIX 2.0 requires UUIDv4; 2.1 only requires the RFC 4122
+    variant.
+
+    :param uuid_str: A UUID as a string
+    :param spec_version: The STIX spec version
+    :return: True if the UUID is valid, False if not
+    :raises ValueError: If uuid_str is malformed
+    """
+    if interoperability:
+        return ID_REGEX_interoperability.match(uuid_str)
+
+    uuid_obj = uuid.UUID(uuid_str)
+
+    ok = uuid_obj.variant == uuid.RFC_4122
+    if ok and spec_version == "2.0":
+        ok = uuid_obj.version == 4
+
+    return ok
+
+
+def _validate_id(id_, spec_version, required_prefix, interoperability):
+    """
+    Check the STIX identifier for correctness, raise an exception if there are
+    errors.
+
+    :param id_: The STIX identifier
+    :param spec_version: The STIX specification version to use
+    :param required_prefix: The required prefix on the identifier, if any.
+        This function doesn't add a "--" suffix to the prefix, so callers must
+        add it if it is important.  Pass None to skip the prefix check.
+    :raises ValueError: If there are any errors with the identifier
+    """
+    if required_prefix:
+        if not id_.startswith(required_prefix):
+            raise ValueError("must start with '{}'.".format(required_prefix))
+
+    try:
+        if required_prefix:
+            uuid_part = id_[len(required_prefix):]
+        else:
+            idx = id_.index("--")
+            uuid_part = id_[idx+2:]
+
+        result = _check_uuid(uuid_part, spec_version, interoperability)
+    except ValueError:
+        # replace their ValueError with ours
+        raise ValueError(ERROR_INVALID_ID.format(id_))
+
+    if not result:
+        raise ValueError(ERROR_INVALID_ID.format(id_))
+
+
 class Property(object):
     """Represent a property of STIX data type.
 
@@ -68,7 +114,7 @@ class Property(object):
         - Return a value that is valid for this property. If ``value`` is not
           valid for this property, this will attempt to transform it first. If
           ``value`` is not valid and no such transformation is possible, it
-          should raise a ValueError.
+          should raise an exception.
     - ``def default(self):``
         - provide a default value for this property.
         - ``default()`` can return the special value ``NOW`` to use the current
@@ -206,19 +252,14 @@ class TypeProperty(Property):
 
 class IDProperty(Property):
 
-    def __init__(self, type):
+    def __init__(self, type, spec_version=stix2.DEFAULT_VERSION):
         self.required_prefix = type + "--"
+        self.spec_version = spec_version
         super(IDProperty, self).__init__()
 
     def clean(self, value):
-        if not value.startswith(self.required_prefix):
-            raise ValueError("must start with '{}'.".format(self.required_prefix))
-        if hasattr(self, 'interoperability') and self.interoperability:
-            if not ID_REGEX_interoperability.match(value):
-                raise ValueError(ERROR_INVALID_ID)
-        else:
-            if not ID_REGEX.match(value):
-                raise ValueError(ERROR_INVALID_ID)
+        interoperability = self.interoperability if hasattr(self, 'interoperability') and self.interoperability else False
+        _validate_id(value, self.spec_version, self.required_prefix, interoperability)
         return value
 
     def default(self):
@@ -307,7 +348,7 @@ class TimestampProperty(Property):
 
 class DictionaryProperty(Property):
 
-    def __init__(self, spec_version='2.0', **kwargs):
+    def __init__(self, spec_version=stix2.DEFAULT_VERSION, **kwargs):
         self.spec_version = spec_version
         super(DictionaryProperty, self).__init__(**kwargs)
 
@@ -389,26 +430,57 @@ class HexProperty(Property):
 
 class ReferenceProperty(Property):
 
-    def __init__(self, type=None, **kwargs):
+    def __init__(self, valid_types=None, invalid_types=None, spec_version=stix2.DEFAULT_VERSION, **kwargs):
         """
         references sometimes must be to a specific object type
         """
-        self.type = type
+        self.spec_version = spec_version
+
+        # These checks need to be done prior to the STIX object finishing construction
+        # and thus we can't use base.py's _check_mutually_exclusive_properties()
+        # in the typical location of _check_object_constraints() in sdo.py
+        if valid_types and invalid_types:
+            raise MutuallyExclusivePropertiesError(self.__class__, ['invalid_types', 'valid_types'])
+        elif valid_types is None and invalid_types is None:
+            raise MissingPropertiesError(self.__class__, ['invalid_types', 'valid_types'])
+
+        if valid_types and type(valid_types) is not list:
+            valid_types = [valid_types]
+        elif invalid_types and type(invalid_types) is not list:
+            invalid_types = [invalid_types]
+
+        self.valid_types = valid_types
+        self.invalid_types = invalid_types
+
         super(ReferenceProperty, self).__init__(**kwargs)
 
     def clean(self, value):
         if isinstance(value, _STIXBase):
             value = value.id
         value = str(value)
-        if self.type:
-            if not value.startswith(self.type):
-                raise ValueError("must start with '{}'.".format(self.type))
-        if hasattr(self, 'interoperability') and self.interoperability:
-            if not ID_REGEX_interoperability.match(value):
-                raise ValueError(ERROR_INVALID_ID)
-        else:
-            if not ID_REGEX.match(value):
-                raise ValueError(ERROR_INVALID_ID)
+
+        possible_prefix = value[:value.index('--') + 2]
+
+        if self.valid_types:
+            if self.valid_types == ["only_SDO"]:
+                self.valid_types = STIX2_OBJ_MAPS['v21']['objects'].keys()
+            elif self.valid_types == ["only_SCO"]:
+                self.valid_types = STIX2_OBJ_MAPS['v21']['observables'].keys()
+            elif self.valid_types == ["only_SCO_&_SRO"]:
+                self.valid_types = list(STIX2_OBJ_MAPS['v21']['observables'].keys()) + ['relationship', 'sighting']
+
+            if possible_prefix[:-2] in self.valid_types:
+                required_prefix = possible_prefix
+            else:
+                raise ValueError("The type-specifying prefix '%s' for this property is not valid" % (possible_prefix))
+        elif self.invalid_types:
+            if possible_prefix[:-2] not in self.invalid_types:
+                required_prefix = possible_prefix
+            else:
+                raise ValueError("An invalid type-specifying prefix '%s' was specified for this property" % (possible_prefix, value))
+        interoperability = self.interoperability if hasattr(self, 'interoperability') and self.interoperability else False
+        _validate_id(value, self.spec_version, required_prefix)
+
         return value
 
 
@@ -477,7 +549,7 @@ class ObservableProperty(Property):
     """Property for holding Cyber Observable Objects.
     """
 
-    def __init__(self, spec_version='2.0', allow_custom=False, *args, **kwargs):
+    def __init__(self, spec_version=stix2.DEFAULT_VERSION, allow_custom=False, *args, **kwargs):
         self.allow_custom = allow_custom
         self.spec_version = spec_version
         super(ObservableProperty, self).__init__(*args, **kwargs)
@@ -512,7 +584,7 @@ class ExtensionsProperty(DictionaryProperty):
     """Property for representing extensions on Observable objects.
     """
 
-    def __init__(self, spec_version='2.0', allow_custom=False, enclosing_type=None, required=False):
+    def __init__(self, spec_version=stix2.DEFAULT_VERSION, allow_custom=False, enclosing_type=None, required=False):
         self.allow_custom = allow_custom
         self.enclosing_type = enclosing_type
         super(ExtensionsProperty, self).__init__(spec_version=spec_version, required=required)
@@ -545,13 +617,16 @@ class ExtensionsProperty(DictionaryProperty):
                 else:
                     raise ValueError("Cannot determine extension type.")
             else:
-                raise CustomContentError("Can't parse unknown extension type: {}".format(key))
+                if self.allow_custom:
+                    dictified[key] = subvalue
+                else:
+                    raise CustomContentError("Can't parse unknown extension type: {}".format(key))
         return dictified
 
 
 class STIXObjectProperty(Property):
 
-    def __init__(self, spec_version='2.0', allow_custom=False, interoperability=False, *args, **kwargs):
+    def __init__(self, spec_version=stix2.DEFAULT_VERSION, allow_custom=False, interoperability=False, *args, **kwargs):
         self.allow_custom = allow_custom
         self.spec_version = spec_version
         self.interoperability = interoperability

stix2/test/test_workbench.py

@@ -1,37 +1,43 @@
+import importlib
 import os
 
 import stix2
 from stix2.workbench import (
-    AttackPattern, Campaign, CourseOfAction, ExternalReference,
-    FileSystemSource, Filter, Identity, Indicator, IntrusionSet, Malware,
-    MarkingDefinition, ObservedData, Relationship, Report, StatementMarking,
-    ThreatActor, Tool, Vulnerability, add_data_source, all_versions,
-    attack_patterns, campaigns, courses_of_action, create, get, identities,
-    indicators, intrusion_sets, malware, observed_data, query, reports, save,
-    set_default_created, set_default_creator, set_default_external_refs,
+    _STIX_VID, AttackPattern, Bundle, Campaign, CourseOfAction,
+    ExternalReference, File, FileSystemSource, Filter, Identity, Indicator,
+    IntrusionSet, Malware, MarkingDefinition, NTFSExt, ObservedData,
+    Relationship, Report, StatementMarking, ThreatActor, Tool, Vulnerability,
+    add_data_source, all_versions, attack_patterns, campaigns,
+    courses_of_action, create, get, identities, indicators, intrusion_sets,
+    malware, observed_data, query, reports, save, set_default_created,
+    set_default_creator, set_default_external_refs,
     set_default_object_marking_refs, threat_actors, tools, vulnerabilities,
 )
 
-from .constants import (
-    ATTACK_PATTERN_ID, ATTACK_PATTERN_KWARGS, CAMPAIGN_ID, CAMPAIGN_KWARGS,
-    COURSE_OF_ACTION_ID, COURSE_OF_ACTION_KWARGS, IDENTITY_ID, IDENTITY_KWARGS,
-    INDICATOR_ID, INDICATOR_KWARGS, INTRUSION_SET_ID, INTRUSION_SET_KWARGS,
-    MALWARE_ID, MALWARE_KWARGS, OBSERVED_DATA_ID, OBSERVED_DATA_KWARGS,
-    REPORT_ID, REPORT_KWARGS, THREAT_ACTOR_ID, THREAT_ACTOR_KWARGS, TOOL_ID,
-    TOOL_KWARGS, VULNERABILITY_ID, VULNERABILITY_KWARGS,
+# Auto-detect some settings based on the current default STIX version
+_STIX_DATA_PATH = os.path.join(
+    os.path.dirname(os.path.realpath(__file__)),
+    _STIX_VID,
+    "stix2_data",
 )
+_STIX_CONSTANTS_MODULE = "stix2.test." + _STIX_VID + ".constants"
+
+
+constants = importlib.import_module(_STIX_CONSTANTS_MODULE)
 
 
 def test_workbench_environment():
 
     # Create a STIX object
-    ind = create(Indicator, id=INDICATOR_ID, **INDICATOR_KWARGS)
+    ind = create(
+        Indicator, id=constants.INDICATOR_ID, **constants.INDICATOR_KWARGS
+    )
     save(ind)
 
-    resp = get(INDICATOR_ID)
+    resp = get(constants.INDICATOR_ID)
     assert resp['labels'][0] == 'malicious-activity'
 
-    resp = all_versions(INDICATOR_ID)
+    resp = all_versions(constants.INDICATOR_ID)
     assert len(resp) == 1
 
     # Search on something other than id
@@ -41,176 +47,193 @@ def test_workbench_environment():
 
 
 def test_workbench_get_all_attack_patterns():
-    mal = AttackPattern(id=ATTACK_PATTERN_ID, **ATTACK_PATTERN_KWARGS)
+    mal = AttackPattern(
+        id=constants.ATTACK_PATTERN_ID, **constants.ATTACK_PATTERN_KWARGS
+    )
     save(mal)
 
     resp = attack_patterns()
     assert len(resp) == 1
-    assert resp[0].id == ATTACK_PATTERN_ID
+    assert resp[0].id == constants.ATTACK_PATTERN_ID
 
 
 def test_workbench_get_all_campaigns():
-    cam = Campaign(id=CAMPAIGN_ID, **CAMPAIGN_KWARGS)
+    cam = Campaign(id=constants.CAMPAIGN_ID, **constants.CAMPAIGN_KWARGS)
     save(cam)
 
     resp = campaigns()
     assert len(resp) == 1
-    assert resp[0].id == CAMPAIGN_ID
+    assert resp[0].id == constants.CAMPAIGN_ID
 
 
 def test_workbench_get_all_courses_of_action():
-    coa = CourseOfAction(id=COURSE_OF_ACTION_ID, **COURSE_OF_ACTION_KWARGS)
+    coa = CourseOfAction(
+        id=constants.COURSE_OF_ACTION_ID, **constants.COURSE_OF_ACTION_KWARGS
+    )
     save(coa)
 
     resp = courses_of_action()
     assert len(resp) == 1
-    assert resp[0].id == COURSE_OF_ACTION_ID
+    assert resp[0].id == constants.COURSE_OF_ACTION_ID
 
 
 def test_workbench_get_all_identities():
-    idty = Identity(id=IDENTITY_ID, **IDENTITY_KWARGS)
+    idty = Identity(id=constants.IDENTITY_ID, **constants.IDENTITY_KWARGS)
     save(idty)
 
     resp = identities()
     assert len(resp) == 1
-    assert resp[0].id == IDENTITY_ID
+    assert resp[0].id == constants.IDENTITY_ID
 
 
 def test_workbench_get_all_indicators():
     resp = indicators()
     assert len(resp) == 1
-    assert resp[0].id == INDICATOR_ID
+    assert resp[0].id == constants.INDICATOR_ID
 
 
 def test_workbench_get_all_intrusion_sets():
-    ins = IntrusionSet(id=INTRUSION_SET_ID, **INTRUSION_SET_KWARGS)
+    ins = IntrusionSet(
+        id=constants.INTRUSION_SET_ID, **constants.INTRUSION_SET_KWARGS
+    )
     save(ins)
 
     resp = intrusion_sets()
     assert len(resp) == 1
-    assert resp[0].id == INTRUSION_SET_ID
+    assert resp[0].id == constants.INTRUSION_SET_ID
 
 
 def test_workbench_get_all_malware():
-    mal = Malware(id=MALWARE_ID, **MALWARE_KWARGS)
+    mal = Malware(id=constants.MALWARE_ID, **constants.MALWARE_KWARGS)
     save(mal)
 
     resp = malware()
     assert len(resp) == 1
-    assert resp[0].id == MALWARE_ID
+    assert resp[0].id == constants.MALWARE_ID
 
 
 def test_workbench_get_all_observed_data():
-    od = ObservedData(id=OBSERVED_DATA_ID, **OBSERVED_DATA_KWARGS)
+    od = ObservedData(
+        id=constants.OBSERVED_DATA_ID, **constants.OBSERVED_DATA_KWARGS
+    )
     save(od)
 
     resp = observed_data()
     assert len(resp) == 1
-    assert resp[0].id == OBSERVED_DATA_ID
+    assert resp[0].id == constants.OBSERVED_DATA_ID
 
 
 def test_workbench_get_all_reports():
-    rep = Report(id=REPORT_ID, **REPORT_KWARGS)
+    rep = Report(id=constants.REPORT_ID, **constants.REPORT_KWARGS)
     save(rep)
 
     resp = reports()
     assert len(resp) == 1
-    assert resp[0].id == REPORT_ID
+    assert resp[0].id == constants.REPORT_ID
 
 
 def test_workbench_get_all_threat_actors():
-    thr = ThreatActor(id=THREAT_ACTOR_ID, **THREAT_ACTOR_KWARGS)
+    thr = ThreatActor(
+        id=constants.THREAT_ACTOR_ID, **constants.THREAT_ACTOR_KWARGS
+    )
     save(thr)
 
     resp = threat_actors()
     assert len(resp) == 1
-    assert resp[0].id == THREAT_ACTOR_ID
+    assert resp[0].id == constants.THREAT_ACTOR_ID
 
 
 def test_workbench_get_all_tools():
-    tool = Tool(id=TOOL_ID, **TOOL_KWARGS)
+    tool = Tool(id=constants.TOOL_ID, **constants.TOOL_KWARGS)
     save(tool)
 
     resp = tools()
     assert len(resp) == 1
-    assert resp[0].id == TOOL_ID
+    assert resp[0].id == constants.TOOL_ID
 
 
 def test_workbench_get_all_vulnerabilities():
-    vuln = Vulnerability(id=VULNERABILITY_ID, **VULNERABILITY_KWARGS)
+    vuln = Vulnerability(
+        id=constants.VULNERABILITY_ID, **constants.VULNERABILITY_KWARGS
+    )
     save(vuln)
 
     resp = vulnerabilities()
     assert len(resp) == 1
-    assert resp[0].id == VULNERABILITY_ID
+    assert resp[0].id == constants.VULNERABILITY_ID
 
 
 def test_workbench_add_to_bundle():
-    vuln = Vulnerability(**VULNERABILITY_KWARGS)
-    bundle = stix2.v20.Bundle(vuln)
+    vuln = Vulnerability(**constants.VULNERABILITY_KWARGS)
+    bundle = Bundle(vuln)
     assert bundle.objects[0].name == 'Heartbleed'
 
 
 def test_workbench_relationships():
-    rel = Relationship(INDICATOR_ID, 'indicates', MALWARE_ID)
+    rel = Relationship(
+        constants.INDICATOR_ID, 'indicates', constants.MALWARE_ID,
+    )
     save(rel)
 
-    ind = get(INDICATOR_ID)
+    ind = get(constants.INDICATOR_ID)
     resp = ind.relationships()
     assert len(resp) == 1
     assert resp[0].relationship_type == 'indicates'
-    assert resp[0].source_ref == INDICATOR_ID
-    assert resp[0].target_ref == MALWARE_ID
+    assert resp[0].source_ref == constants.INDICATOR_ID
+    assert resp[0].target_ref == constants.MALWARE_ID
 
 
 def test_workbench_created_by():
-    intset = IntrusionSet(name="Breach 123", created_by_ref=IDENTITY_ID)
+    intset = IntrusionSet(
+        name="Breach 123", created_by_ref=constants.IDENTITY_ID,
+    )
     save(intset)
     creator = intset.created_by()
-    assert creator.id == IDENTITY_ID
+    assert creator.id == constants.IDENTITY_ID
 
 
 def test_workbench_related():
-    rel1 = Relationship(MALWARE_ID, 'targets', IDENTITY_ID)
-    rel2 = Relationship(CAMPAIGN_ID, 'uses', MALWARE_ID)
+    rel1 = Relationship(constants.MALWARE_ID, 'targets', constants.IDENTITY_ID)
+    rel2 = Relationship(constants.CAMPAIGN_ID, 'uses', constants.MALWARE_ID)
     save([rel1, rel2])
 
-    resp = get(MALWARE_ID).related()
+    resp = get(constants.MALWARE_ID).related()
     assert len(resp) == 3
-    assert any(x['id'] == CAMPAIGN_ID for x in resp)
-    assert any(x['id'] == INDICATOR_ID for x in resp)
-    assert any(x['id'] == IDENTITY_ID for x in resp)
+    assert any(x['id'] == constants.CAMPAIGN_ID for x in resp)
+    assert any(x['id'] == constants.INDICATOR_ID for x in resp)
+    assert any(x['id'] == constants.IDENTITY_ID for x in resp)
 
-    resp = get(MALWARE_ID).related(relationship_type='indicates')
+    resp = get(constants.MALWARE_ID).related(relationship_type='indicates')
     assert len(resp) == 1
 
 
 def test_workbench_related_with_filters():
-    malware = Malware(labels=["ransomware"], name="CryptorBit", created_by_ref=IDENTITY_ID)
-    rel = Relationship(malware.id, 'variant-of', MALWARE_ID)
+    malware = Malware(
+        labels=["ransomware"], name="CryptorBit", created_by_ref=constants.IDENTITY_ID,
+    )
+    rel = Relationship(malware.id, 'variant-of', constants.MALWARE_ID)
     save([malware, rel])
 
-    filters = [Filter('created_by_ref', '=', IDENTITY_ID)]
-    resp = get(MALWARE_ID).related(filters=filters)
+    filters = [Filter('created_by_ref', '=', constants.IDENTITY_ID)]
+    resp = get(constants.MALWARE_ID).related(filters=filters)
 
     assert len(resp) == 1
     assert resp[0].name == malware.name
-    assert resp[0].created_by_ref == IDENTITY_ID
+    assert resp[0].created_by_ref == constants.IDENTITY_ID
 
     # filters arg can also be single filter
-    resp = get(MALWARE_ID).related(filters=filters[0])
+    resp = get(constants.MALWARE_ID).related(filters=filters[0])
     assert len(resp) == 1
 
 
 def test_add_data_source():
-    fs_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "stix2_data")
-    fs = FileSystemSource(fs_path)
+    fs = FileSystemSource(_STIX_DATA_PATH)
     add_data_source(fs)
 
     resp = tools()
     assert len(resp) == 3
     resp_ids = [tool.id for tool in resp]
-    assert TOOL_ID in resp_ids
+    assert constants.TOOL_ID in resp_ids
     assert 'tool--03342581-f790-4f03-ba41-e82e67392e23' in resp_ids
     assert 'tool--242f3da3-4425-4d11-8f5c-b842886da966' in resp_ids
 
@@ -229,22 +252,28 @@ def test_additional_filters_list():
 
 
 def test_default_creator():
-    set_default_creator(IDENTITY_ID)
-    campaign = Campaign(**CAMPAIGN_KWARGS)
+    set_default_creator(constants.IDENTITY_ID)
+    campaign = Campaign(**constants.CAMPAIGN_KWARGS)
 
-    assert 'created_by_ref' not in CAMPAIGN_KWARGS
-    assert campaign.created_by_ref == IDENTITY_ID
+    assert 'created_by_ref' not in constants.CAMPAIGN_KWARGS
+    assert campaign.created_by_ref == constants.IDENTITY_ID
+
+    # turn off side-effects to avoid affecting future tests
+    set_default_creator(None)
 
 
 def test_default_created_timestamp():
     timestamp = "2018-03-19T01:02:03.000Z"
     set_default_created(timestamp)
-    campaign = Campaign(**CAMPAIGN_KWARGS)
+    campaign = Campaign(**constants.CAMPAIGN_KWARGS)
 
-    assert 'created' not in CAMPAIGN_KWARGS
+    assert 'created' not in constants.CAMPAIGN_KWARGS
     assert stix2.utils.format_datetime(campaign.created) == timestamp
     assert stix2.utils.format_datetime(campaign.modified) == timestamp
 
+    # turn off side-effects to avoid affecting future tests
+    set_default_created(None)
+
 
 def test_default_external_refs():
     ext_ref = ExternalReference(
@@ -252,11 +281,14 @@ def test_default_external_refs():
         description="Threat report",
     )
     set_default_external_refs(ext_ref)
-    campaign = Campaign(**CAMPAIGN_KWARGS)
+    campaign = Campaign(**constants.CAMPAIGN_KWARGS)
 
     assert campaign.external_references[0].source_name == "ACME Threat Intel"
     assert campaign.external_references[0].description == "Threat report"
 
+    # turn off side-effects to avoid affecting future tests
+    set_default_external_refs([])
+
 
 def test_default_object_marking_refs():
     stmt_marking = StatementMarking("Copyright 2016, Example Corp")
@@ -265,18 +297,21 @@ def test_default_object_marking_refs():
         definition=stmt_marking,
     )
     set_default_object_marking_refs(mark_def)
-    campaign = Campaign(**CAMPAIGN_KWARGS)
+    campaign = Campaign(**constants.CAMPAIGN_KWARGS)
 
     assert campaign.object_marking_refs[0] == mark_def.id
 
+    # turn off side-effects to avoid affecting future tests
+    set_default_object_marking_refs([])
+
 
 def test_workbench_custom_property_object_in_observable_extension():
-    ntfs = stix2.v20.NTFSExt(
+    ntfs = NTFSExt(
         allow_custom=True,
         sid=1,
         x_foo='bar',
     )
-    artifact = stix2.v20.File(
+    artifact = File(
         name='test',
         extensions={'ntfs-ext': ntfs},
     )
@@ -293,7 +328,7 @@ def test_workbench_custom_property_object_in_observable_extension():
 
 
 def test_workbench_custom_property_dict_in_observable_extension():
-    artifact = stix2.v20.File(
+    artifact = File(
         allow_custom=True,
         name='test',
         extensions={

stix2/test/v20/test_bundle.py

@@ -4,6 +4,7 @@ import pytest
 
 import stix2
 
+from ...exceptions import InvalidValueError
 from .constants import IDENTITY_ID
 
 EXPECTED_BUNDLE = """{
@@ -156,15 +157,15 @@ def test_create_bundle_with_arg_listarg_and_kwarg(indicator, malware, relationsh
 
 
 def test_create_bundle_invalid(indicator, malware, relationship):
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.v20.Bundle(objects=[1])
     assert excinfo.value.reason == "This property may only contain a dictionary or object"
 
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.v20.Bundle(objects=[{}])
     assert excinfo.value.reason == "This property may only contain a non-empty dictionary or object"
 
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.v20.Bundle(objects=[{'type': 'bundle'}])
     assert excinfo.value.reason == 'This property may not contain a Bundle object'
 
@@ -175,7 +176,7 @@ def test_parse_bundle(version):
 
     assert bundle.type == "bundle"
     assert bundle.id.startswith("bundle--")
-    assert type(bundle.objects[0]) is stix2.v20.Indicator
+    assert isinstance(bundle.objects[0], stix2.v20.Indicator)
     assert bundle.objects[0].type == 'indicator'
     assert bundle.objects[1].type == 'malware'
     assert bundle.objects[2].type == 'relationship'
@@ -232,7 +233,7 @@ def test_bundle_with_different_spec_objects():
         },
     ]
 
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.v20.Bundle(objects=data)
 
     assert "Spec version 2.0 bundles don't yet support containing objects of a different spec version." in str(excinfo.value)

stix2/test/v20/test_core.py

@@ -74,7 +74,9 @@ def test_register_object_with_version():
     v = 'v20'
 
     assert bundle.objects[0].type in core.STIX2_OBJ_MAPS[v]['objects']
-    assert v in str(bundle.objects[0].__class__)
+    # spec_version is not in STIX 2.0, and is required in 2.1, so this
+    # suffices as a test for a STIX 2.0 object.
+    assert "spec_version" not in bundle.objects[0]
 
 
 def test_register_marking_with_version():

stix2/test/v20/test_custom.py

@@ -1,7 +1,9 @@
 import pytest
 
 import stix2
+import stix2.v20
 
+from ...exceptions import InvalidValueError
 from .constants import FAKE_TIME, IDENTITY_ID, MARKING_DEFINITION_ID
 
 IDENTITY_CUSTOM_PROP = stix2.v20.Identity(
@@ -95,7 +97,7 @@ def test_identity_custom_property_allowed():
 def test_parse_identity_custom_property(data):
     with pytest.raises(stix2.exceptions.ExtraPropertiesError) as excinfo:
         stix2.parse(data, version="2.0")
-    assert excinfo.value.cls == stix2.v20.Identity
+    assert issubclass(excinfo.value.cls, stix2.v20.Identity)
     assert excinfo.value.properties == ['foo']
     assert "Unexpected properties for" in str(excinfo.value)
 
@@ -133,7 +135,7 @@ def test_custom_property_dict_in_bundled_object():
         'identity_class': 'individual',
         'x_foo': 'bar',
     }
-    with pytest.raises(stix2.exceptions.ExtraPropertiesError):
+    with pytest.raises(InvalidValueError):
         stix2.v20.Bundle(custom_identity)
 
     bundle = stix2.v20.Bundle(custom_identity, allow_custom=True)
@@ -199,7 +201,7 @@ def test_custom_property_object_in_observable_extension():
 
 
 def test_custom_property_dict_in_observable_extension():
-    with pytest.raises(stix2.exceptions.ExtraPropertiesError):
+    with pytest.raises(InvalidValueError):
         stix2.v20.File(
             name='test',
             extensions={
@@ -718,7 +720,7 @@ def test_custom_extension():
 def test_custom_extension_wrong_observable_type():
     # NewExtension is an extension of DomainName, not File
     ext = NewExtension(property1='something')
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.v20.File(
             name="abc.txt",
             extensions={
@@ -884,6 +886,49 @@ def test_parse_observable_with_custom_extension():
     assert parsed.extensions['x-new-ext'].property2 == 12
 
 
+def test_custom_and_spec_extension_mix():
+    """
+    Try to make sure that when allow_custom=True, encountering a custom
+    extension doesn't result in a completely uncleaned extensions property.
+    """
+
+    file_obs = stix2.v20.File(
+        name="my_file.dat",
+        extensions={
+            "x-custom1": {
+                "a": 1,
+                "b": 2,
+            },
+            "ntfs-ext": {
+                "sid": "S-1-whatever",
+            },
+            "x-custom2": {
+                "z": 99.9,
+                "y": False,
+            },
+            "raster-image-ext": {
+                "image_height": 1024,
+                "image_width": 768,
+                "bits_per_pixel": 32,
+            },
+        },
+        allow_custom=True,
+    )
+
+    assert file_obs.extensions["x-custom1"] == {"a": 1, "b": 2}
+    assert file_obs.extensions["x-custom2"] == {"y": False, "z": 99.9}
+    assert file_obs.extensions["ntfs-ext"].sid == "S-1-whatever"
+    assert file_obs.extensions["raster-image-ext"].image_height == 1024
+
+    # Both of these should have been converted to objects, not left as dicts.
+    assert isinstance(
+        file_obs.extensions["raster-image-ext"], stix2.v20.RasterImageExt,
+    )
+    assert isinstance(
+        file_obs.extensions["ntfs-ext"], stix2.v20.NTFSExt,
+    )
+
+
 @pytest.mark.parametrize(
     "data", [
         # URL is not in EXT_MAP
@@ -911,7 +956,7 @@ def test_parse_observable_with_custom_extension():
     ],
 )
 def test_parse_observable_with_unregistered_custom_extension(data):
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.parse_observable(data, version='2.0')
     assert "Can't parse unknown extension type" in str(excinfo.value)
 

stix2/test/v20/test_granular_markings.py

@@ -2,7 +2,7 @@
 import pytest
 
 from stix2 import markings
-from stix2.exceptions import MarkingNotFoundError
+from stix2.exceptions import InvalidSelectorError, MarkingNotFoundError
 from stix2.v20 import TLP_RED, Malware
 
 from .constants import MALWARE_MORE_KWARGS as MALWARE_KWARGS_CONST
@@ -179,7 +179,7 @@ def test_add_marking_mark_same_property_same_marking():
     ],
 )
 def test_add_marking_bad_selector(data, marking):
-    with pytest.raises(AssertionError):
+    with pytest.raises(InvalidSelectorError):
         markings.add_markings(data, marking[0], marking[1])
 
 
@@ -299,7 +299,7 @@ def test_get_markings_multiple_selectors(data):
 )
 def test_get_markings_bad_selector(data, selector):
     """Test bad selectors raise exception"""
-    with pytest.raises(AssertionError):
+    with pytest.raises(InvalidSelectorError):
         markings.get_markings(data, selector)
 
 
@@ -560,7 +560,7 @@ def test_remove_marking_bad_selector():
     before = {
         "description": "test description",
     }
-    with pytest.raises(AssertionError):
+    with pytest.raises(InvalidSelectorError):
         markings.remove_markings(before, ["marking-definition--1", "marking-definition--2"], ["title"])
 
 
@@ -642,7 +642,7 @@ def test_is_marked_smoke(data):
 )
 def test_is_marked_invalid_selector(data, selector):
     """Test invalid selector raises an error."""
-    with pytest.raises(AssertionError):
+    with pytest.raises(InvalidSelectorError):
         markings.is_marked(data, selectors=selector)
 
 
@@ -836,7 +836,7 @@ def test_is_marked_positional_arguments_combinations():
 
 
 def test_create_sdo_with_invalid_marking():
-    with pytest.raises(AssertionError) as excinfo:
+    with pytest.raises(InvalidSelectorError) as excinfo:
         Malware(
             granular_markings=[
                 {
@@ -974,7 +974,7 @@ def test_set_marking_bad_selector(marking):
         **MALWARE_KWARGS
     )
 
-    with pytest.raises(AssertionError):
+    with pytest.raises(InvalidSelectorError):
         before = markings.set_markings(before, marking[0], marking[1])
 
     assert before == after
@@ -1080,7 +1080,7 @@ def test_clear_marking_all_selectors(data):
 )
 def test_clear_marking_bad_selector(data, selector):
     """Test bad selector raises exception."""
-    with pytest.raises(AssertionError):
+    with pytest.raises(InvalidSelectorError):
         markings.clear_markings(data, selector)
 
 

stix2/test/v20/test_indicator.py

@@ -112,8 +112,6 @@ def test_indicator_created_ref_invalid_format():
 
     assert excinfo.value.cls == stix2.v20.Indicator
     assert excinfo.value.prop_name == "created_by_ref"
-    assert excinfo.value.reason == "must start with 'identity'."
-    assert str(excinfo.value) == "Invalid value for Indicator 'created_by_ref': must start with 'identity'."
 
 
 def test_indicator_revoked_invalid():

stix2/test/v20/test_malware.py

@@ -6,6 +6,7 @@ import pytz
 
 import stix2
 
+from ...exceptions import InvalidValueError
 from .constants import FAKE_TIME, MALWARE_ID, MALWARE_KWARGS
 
 EXPECTED_MALWARE = """{
@@ -145,7 +146,7 @@ def test_parse_malware(data):
 
 def test_parse_malware_invalid_labels():
     data = re.compile('\\[.+\\]', re.DOTALL).sub('1', EXPECTED_MALWARE)
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.parse(data, version="2.0")
     assert "Invalid value for Malware 'labels'" in str(excinfo.value)
 

stix2/test/v20/test_object_markings.py

@@ -4,6 +4,7 @@ import pytest
 from stix2 import exceptions, markings
 from stix2.v20 import TLP_AMBER, Malware
 
+from ...exceptions import MarkingNotFoundError
 from .constants import FAKE_TIME, MALWARE_ID
 from .constants import MALWARE_KWARGS as MALWARE_KWARGS_CONST
 from .constants import MARKING_IDS
@@ -350,7 +351,7 @@ def test_remove_markings_bad_markings():
         object_marking_refs=[MARKING_IDS[0], MARKING_IDS[1], MARKING_IDS[2]],
         **MALWARE_KWARGS
     )
-    with pytest.raises(AssertionError) as excinfo:
+    with pytest.raises(MarkingNotFoundError) as excinfo:
         markings.remove_markings(before, [MARKING_IDS[4]], None)
     assert str(excinfo.value) == "Marking ['%s'] was not found in Malware!" % MARKING_IDS[4]
 

stix2/test/v20/test_observed_data.py

@@ -6,6 +6,7 @@ import pytz
 
 import stix2
 
+from ...exceptions import InvalidValueError
 from .constants import IDENTITY_ID, OBSERVED_DATA_ID
 
 OBJECTS_REGEX = re.compile('\"objects\": {(?:.*?)(?:(?:[^{]*?)|(?:{[^{]*?}))*}', re.DOTALL)
@@ -239,7 +240,7 @@ def test_parse_artifact_valid(data):
 )
 def test_parse_artifact_invalid(data):
     odata_str = OBJECTS_REGEX.sub('"objects": { %s }' % data, EXPECTED)
-    with pytest.raises(ValueError):
+    with pytest.raises(InvalidValueError):
         stix2.parse(odata_str, version="2.0")
 
 
@@ -468,11 +469,10 @@ def test_parse_email_message_with_at_least_one_error(data):
         "4": "artifact",
         "5": "file",
     }
-    with pytest.raises(stix2.exceptions.AtLeastOnePropertyError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.parse_observable(data, valid_refs, version='2.0')
 
-    assert excinfo.value.cls == stix2.v20.EmailMIMEComponent
-    assert excinfo.value.properties == ["body", "body_raw_ref"]
+    assert excinfo.value.cls == stix2.v20.EmailMessage
     assert "At least one of the" in str(excinfo.value)
     assert "must be populated" in str(excinfo.value)
 
@@ -734,7 +734,7 @@ def test_file_example_with_NTFSExt():
 
 
 def test_file_example_with_empty_NTFSExt():
-    with pytest.raises(stix2.exceptions.AtLeastOnePropertyError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.v20.File(
             name="abc.txt",
             extensions={
@@ -742,8 +742,7 @@ def test_file_example_with_empty_NTFSExt():
             },
         )
 
-    assert excinfo.value.cls == stix2.v20.NTFSExt
-    assert excinfo.value.properties == sorted(list(stix2.NTFSExt._properties.keys()))
+    assert excinfo.value.cls == stix2.v20.File
 
 
 def test_file_example_with_PDFExt():
@@ -1112,16 +1111,14 @@ def test_process_example_empty_error():
 
 
 def test_process_example_empty_with_extensions():
-    with pytest.raises(stix2.exceptions.AtLeastOnePropertyError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.v20.Process(
             extensions={
                 "windows-process-ext": {},
             },
         )
 
-    assert excinfo.value.cls == stix2.v20.WindowsProcessExt
-    properties_of_extension = list(stix2.v20.WindowsProcessExt._properties.keys())
-    assert excinfo.value.properties == sorted(properties_of_extension)
+    assert excinfo.value.cls == stix2.v20.Process
 
 
 def test_process_example_windows_process_ext():
@@ -1144,7 +1141,7 @@ def test_process_example_windows_process_ext():
 
 
 def test_process_example_windows_process_ext_empty():
-    with pytest.raises(stix2.exceptions.AtLeastOnePropertyError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.v20.Process(
             pid=1221,
             name="gedit-bin",
@@ -1153,9 +1150,7 @@ def test_process_example_windows_process_ext_empty():
             },
         )
 
-    assert excinfo.value.cls == stix2.v20.WindowsProcessExt
-    properties_of_extension = list(stix2.v20.WindowsProcessExt._properties.keys())
-    assert excinfo.value.properties == sorted(properties_of_extension)
+    assert excinfo.value.cls == stix2.v20.Process
 
 
 def test_process_example_extensions_empty():
@@ -1289,7 +1284,7 @@ def test_user_account_unix_account_ext_example():
 
 
 def test_windows_registry_key_example():
-    with pytest.raises(ValueError):
+    with pytest.raises(InvalidValueError):
         stix2.v20.WindowsRegistryValueType(
             name="Foo",
             data="qwerty",

stix2/test/v20/test_properties.py

@@ -3,9 +3,11 @@ import uuid
 import pytest
 
 import stix2
-from stix2.exceptions import AtLeastOnePropertyError, DictionaryKeyError
+from stix2.exceptions import (
+    AtLeastOnePropertyError, CustomContentError, DictionaryKeyError,
+)
 from stix2.properties import (
-    ERROR_INVALID_ID, BinaryProperty, BooleanProperty, DictionaryProperty,
+    BinaryProperty, BooleanProperty, DictionaryProperty,
     EmbeddedObjectProperty, EnumProperty, ExtensionsProperty, FloatProperty,
     HashesProperty, HexProperty, IDProperty, IntegerProperty, ListProperty,
     Property, ReferenceProperty, STIXObjectProperty, StringProperty,
@@ -89,7 +91,7 @@ def test_type_property():
     assert prop.clean(prop.default())
 
 
-ID_PROP = IDProperty('my-type')
+ID_PROP = IDProperty('my-type', spec_version="2.0")
 MY_ID = 'my-type--232c9d3f-49fc-4440-bb01-607f638778e7'
 
 
@@ -127,7 +129,7 @@ CONSTANT_IDS.extend(constants.RELATIONSHIP_IDS)
 @pytest.mark.parametrize("value", CONSTANT_IDS)
 def test_id_property_valid_for_type(value):
     type = value.split('--', 1)[0]
-    assert IDProperty(type=type).clean(value) == value
+    assert IDProperty(type=type, spec_version="2.0").clean(value) == value
 
 
 def test_id_property_wrong_type():
@@ -147,9 +149,8 @@ def test_id_property_wrong_type():
     ],
 )
 def test_id_property_not_a_valid_hex_uuid(value):
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(ValueError):
         ID_PROP.clean(value)
-    assert str(excinfo.value) == ERROR_INVALID_ID
 
 
 def test_id_property_default():
@@ -275,7 +276,7 @@ def test_boolean_property_invalid(value):
 
 
 def test_reference_property():
-    ref_prop = ReferenceProperty()
+    ref_prop = ReferenceProperty(valid_types="my-type", spec_version="2.0")
 
     assert ref_prop.clean("my-type--00000000-0000-4000-8000-000000000000")
     with pytest.raises(ValueError):
@@ -286,6 +287,16 @@ def test_reference_property():
         ref_prop.clean("my-type--00000000-0000-0000-0000-000000000000")
 
 
+def test_reference_property_specific_type():
+    ref_prop = ReferenceProperty(valid_types="my-type", spec_version="2.0")
+
+    with pytest.raises(ValueError):
+        ref_prop.clean("not-my-type--8a8e8758-f92c-4058-ba38-f061cd42a0cf")
+
+    assert ref_prop.clean("my-type--8a8e8758-f92c-4058-ba38-f061cd42a0cf") == \
+        "my-type--8a8e8758-f92c-4058-ba38-f061cd42a0cf"
+
+
 @pytest.mark.parametrize(
     "value", [
         '2017-01-01T12:34:56Z',
@@ -329,7 +340,7 @@ def test_hex_property():
     ],
 )
 def test_dictionary_property_valid(d):
-    dict_prop = DictionaryProperty()
+    dict_prop = DictionaryProperty(spec_version="2.0")
     assert dict_prop.clean(d)
 
 
@@ -350,7 +361,7 @@ def test_dictionary_property_valid(d):
     ],
 )
 def test_dictionary_property_invalid_key(d):
-    dict_prop = DictionaryProperty()
+    dict_prop = DictionaryProperty(spec_version="2.0")
 
     with pytest.raises(DictionaryKeyError) as excinfo:
         dict_prop.clean(d[0])
@@ -372,7 +383,7 @@ def test_dictionary_property_invalid_key(d):
     ],
 )
 def test_dictionary_property_invalid(d):
-    dict_prop = DictionaryProperty()
+    dict_prop = DictionaryProperty(spec_version="2.0")
 
     with pytest.raises(ValueError) as excinfo:
         dict_prop.clean(d[0])
@@ -382,7 +393,7 @@ def test_dictionary_property_invalid(d):
 def test_property_list_of_dictionary():
     @stix2.v20.CustomObject(
         'x-new-obj', [
-            ('property1', ListProperty(DictionaryProperty(), required=True)),
+            ('property1', ListProperty(DictionaryProperty(spec_version="2.0"), required=True)),
         ],
     )
     class NewObj():
@@ -448,7 +459,7 @@ def test_enum_property_invalid():
 
 
 def test_extension_property_valid():
-    ext_prop = ExtensionsProperty(enclosing_type='file')
+    ext_prop = ExtensionsProperty(spec_version="2.0", enclosing_type='file')
     assert ext_prop({
         'windows-pebinary-ext': {
             'pe_type': 'exe',
@@ -456,23 +467,27 @@ def test_extension_property_valid():
     })
 
 
-@pytest.mark.parametrize(
-    "data", [
-        1,
-        {'foobar-ext': {
-            'pe_type': 'exe',
-        }},
-    ],
-)
-def test_extension_property_invalid(data):
-    ext_prop = ExtensionsProperty(enclosing_type='file')
+def test_extension_property_invalid1():
+    ext_prop = ExtensionsProperty(spec_version="2.0", enclosing_type='file')
     with pytest.raises(ValueError):
-        ext_prop.clean(data)
+        ext_prop.clean(1)
+
+
+def test_extension_property_invalid2():
+    ext_prop = ExtensionsProperty(spec_version="2.0", enclosing_type='file')
+    with pytest.raises(CustomContentError):
+        ext_prop.clean(
+            {
+                'foobar-ext': {
+                    'pe_type': 'exe',
+                },
+            },
+        )
 
 
 def test_extension_property_invalid_type():
-    ext_prop = ExtensionsProperty(enclosing_type='indicator')
-    with pytest.raises(ValueError) as excinfo:
+    ext_prop = ExtensionsProperty(spec_version="2.0", enclosing_type='indicator')
+    with pytest.raises(CustomContentError) as excinfo:
         ext_prop.clean(
             {
                 'windows-pebinary-ext': {

stix2/test/v20/test_report.py

@@ -90,8 +90,6 @@ def test_report_example_objects_in_object_refs_with_bad_id():
 
     assert excinfo.value.cls == stix2.v20.Report
     assert excinfo.value.prop_name == "object_refs"
-    assert excinfo.value.reason == stix2.properties.ERROR_INVALID_ID
-    assert str(excinfo.value) == "Invalid value for Report 'object_refs': " + stix2.properties.ERROR_INVALID_ID
 
 
 @pytest.mark.parametrize(

stix2/test/v20/test_sighting.py

@@ -59,8 +59,6 @@ def test_sighting_bad_where_sighted_refs():
 
     assert excinfo.value.cls == stix2.v20.Sighting
     assert excinfo.value.prop_name == "where_sighted_refs"
-    assert excinfo.value.reason == "must start with 'identity'."
-    assert str(excinfo.value) == "Invalid value for Sighting 'where_sighted_refs': must start with 'identity'."
 
 
 def test_sighting_type_must_be_sightings():
@@ -69,8 +67,6 @@ def test_sighting_type_must_be_sightings():
 
     assert excinfo.value.cls == stix2.v20.Sighting
     assert excinfo.value.prop_name == "type"
-    assert excinfo.value.reason == "must equal 'sighting'."
-    assert str(excinfo.value) == "Invalid value for Sighting 'type': must equal 'sighting'."
 
 
 def test_invalid_kwarg_to_sighting():

stix2/test/v20/test_utils.py

@@ -144,8 +144,8 @@ def test_deduplicate(stix_objs1):
                         "type": "network-traffic",
                         "src_ref": "1",
                         "protocols": [
-                          "tcp",
-                          "http",
+                            "tcp",
+                            "http",
                         ],
                         "extensions": {
                             "http-request-ext": {

stix2/test/v21/conftest.py

@@ -5,7 +5,8 @@ import pytest
 import stix2
 
 from .constants import (
-    FAKE_TIME, INDICATOR_KWARGS, MALWARE_KWARGS, RELATIONSHIP_KWARGS,
+    FAKE_TIME, GROUPING_KWARGS, INDICATOR_KWARGS, INFRASTRUCTURE_KWARGS,
+    MALWARE_KWARGS, RELATIONSHIP_KWARGS,
 )
 
 
@@ -39,6 +40,16 @@ def indicator(uuid4, clock):
     return stix2.v21.Indicator(**INDICATOR_KWARGS)
 
 
+@pytest.fixture
+def infrastructure(uuid4, clock):
+    return stix2.v21.Infrastructure(**INFRASTRUCTURE_KWARGS)
+
+
+@pytest.fixture
+def grouping(uuid4, clock):
+    return stix2.v21.Grouping(**GROUPING_KWARGS)
+
+
 @pytest.fixture
 def malware(uuid4, clock):
     return stix2.v21.Malware(**MALWARE_KWARGS)
@@ -60,6 +71,7 @@ def stix_objs1():
         "modified": "2017-01-27T13:49:53.935Z",
         "name": "Malicious site hosting downloader",
         "pattern": "[url:value = 'http://x4z9arb.cn/4712']",
+        "pattern_type": "stix",
         "spec_version": "2.1",
         "type": "indicator",
         "valid_from": "2017-01-27T13:49:53.935382Z",
@@ -73,6 +85,7 @@ def stix_objs1():
         "modified": "2017-01-27T13:49:53.935Z",
         "name": "Malicious site hosting downloader",
         "pattern": "[url:value = 'http://x4z9arb.cn/4712']",
+        "pattern_type": "stix",
         "spec_version": "2.1",
         "type": "indicator",
         "valid_from": "2017-01-27T13:49:53.935382Z",
@@ -86,6 +99,7 @@ def stix_objs1():
         "modified": "2017-01-27T13:49:53.936Z",
         "name": "Malicious site hosting downloader",
         "pattern": "[url:value = 'http://x4z9arb.cn/4712']",
+        "pattern_type": "stix",
         "spec_version": "2.1",
         "type": "indicator",
         "valid_from": "2017-01-27T13:49:53.935382Z",
@@ -99,6 +113,7 @@ def stix_objs1():
         "modified": "2017-01-27T13:49:53.935Z",
         "name": "Malicious site hosting downloader",
         "pattern": "[url:value = 'http://x4z9arb.cn/4712']",
+        "pattern_type": "stix",
         "spec_version": "2.1",
         "type": "indicator",
         "valid_from": "2017-01-27T13:49:53.935382Z",
@@ -112,6 +127,7 @@ def stix_objs1():
         "modified": "2017-01-27T13:49:53.935Z",
         "name": "Malicious site hosting downloader",
         "pattern": "[url:value = 'http://x4z9arb.cn/4712']",
+        "pattern_type": "stix",
         "spec_version": "2.1",
         "type": "indicator",
         "valid_from": "2017-01-27T13:49:53.935382Z",
@@ -129,6 +145,7 @@ def stix_objs2():
         ],
         "modified": "2017-01-31T13:49:53.935Z",
         "name": "Malicious site hosting downloader",
+        "pattern_type": "stix",
         "pattern": "[url:value = 'http://x4z9arb.cn/4712']",
         "spec_version": "2.1",
         "type": "indicator",
@@ -142,6 +159,7 @@ def stix_objs2():
         ],
         "modified": "2017-01-27T13:49:53.935Z",
         "name": "Malicious site hosting downloader",
+        "pattern_type": "stix",
         "pattern": "[url:value = 'http://x4z9arb.cn/4712']",
         "spec_version": "2.1",
         "type": "indicator",
@@ -155,6 +173,7 @@ def stix_objs2():
         ],
         "modified": "2017-01-27T13:49:53.935Z",
         "name": "Malicious site hosting downloader",
+        "pattern_type": "stix",
         "pattern": "[url:value = 'http://x4z9arb.cn/4712']",
         "spec_version": "2.1",
         "type": "indicator",

stix2/test/v21/constants.py

@@ -7,8 +7,10 @@ FAKE_TIME = dt.datetime(2017, 1, 1, 12, 34, 56, tzinfo=pytz.utc)
 ATTACK_PATTERN_ID = "attack-pattern--0c7b5b88-8ff7-4a4d-aa9d-feb398cd0061"
 CAMPAIGN_ID = "campaign--8e2e2d2b-17d4-4cbf-938f-98ee46b3cd3f"
 COURSE_OF_ACTION_ID = "course-of-action--8e2e2d2b-17d4-4cbf-938f-98ee46b3cd3f"
+GROUPING_ID = "grouping--753abcde-3141-5926-ace5-0a810b1ff996"
 IDENTITY_ID = "identity--311b2d2d-f010-4473-83ec-1edf84858f4c"
 INDICATOR_ID = "indicator--a740531e-63ff-4e49-a9e1-a0a3eed0e3e7"
+INFRASTRUCTURE_ID = "infrastructure--3000ae1b-784c-f03d-8abc-0a625b2ff018"
 INTRUSION_SET_ID = "intrusion-set--4e78f46f-a023-4e5f-bc24-71b3ca22ec29"
 LOCATION_ID = "location--a6e9345f-5a15-4c29-8bb3-7dcc5d168d64"
 MALWARE_ID = "malware--9c4638ec-f1de-4ddb-abf4-1b760417654e"
@@ -70,6 +72,15 @@ COURSE_OF_ACTION_KWARGS = dict(
     name="Block",
 )
 
+GROUPING_KWARGS = dict(
+    name="Harry Potter and the Leet Hackers",
+    context="suspicious-activity",
+    object_refs=[
+        "malware--c8d2fae5-7271-400c-b81d-931a4caf20b9",
+        "identity--988145ed-a3b4-4421-b7a7-273376be67ce",
+    ],
+)
+
 IDENTITY_KWARGS = dict(
     name="John Smith",
     identity_class="individual",
@@ -77,7 +88,14 @@ IDENTITY_KWARGS = dict(
 
 INDICATOR_KWARGS = dict(
     indicator_types=['malicious-activity'],
+    pattern_type="stix",
     pattern="[file:hashes.MD5 = 'd41d8cd98f00b204e9800998ecf8427e']",
+    valid_from="2017-01-01T12:34:56Z",
+)
+
+INFRASTRUCTURE_KWARGS = dict(
+    name="Poison Ivy C2",
+    infrastructure_types=["command-and-control"],
 )
 
 INTRUSION_SET_KWARGS = dict(
@@ -87,6 +105,7 @@ INTRUSION_SET_KWARGS = dict(
 MALWARE_KWARGS = dict(
     malware_types=['ransomware'],
     name="Cryptolocker",
+    is_family=False,
 )
 
 MALWARE_MORE_KWARGS = dict(
@@ -97,6 +116,7 @@ MALWARE_MORE_KWARGS = dict(
     malware_types=['ransomware'],
     name="Cryptolocker",
     description="A ransomware related to ...",
+    is_family=False,
 )
 
 OBSERVED_DATA_KWARGS = dict(

stix2/test/v21/stix2_data/malware/malware--6b616fc1-1505-48e3-8b2c-0d19337bff38.json

@@ -24,5 +24,6 @@
     ],
     "object_marking_refs": [
         "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"
-    ]
+    ],
+    "is_family": false
 }

stix2/test/v21/stix2_data/malware/malware--6b616fc1-1505-48e3-8b2c-0d19337bff38/20170531213258226477.json

@@ -27,7 +27,8 @@
                 "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"
             ],
             "spec_version": "2.1",
-            "type": "malware"
+            "type": "malware",
+            "is_family": false
         }
     ],
     "type": "bundle"

stix2/test/v21/stix2_data/malware/malware--6b616fc1-1505-48e3-8b2c-0d19337bff38/20181101232448456000.json

@@ -24,5 +24,6 @@
     ],
     "object_marking_refs": [
         "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"
-    ]
+    ],
+    "is_family": false
 }

stix2/test/v21/stix2_data/malware/malware--6b616fc1-1505-48e3-8b2c-0d19337bff38/20181101232448457000.json

@@ -24,5 +24,6 @@
     ],
     "object_marking_refs": [
         "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"
-    ]
+    ],
+    "is_family": false
 }

stix2/test/v21/stix2_data/malware/malware--92ec0cbd-2c30-44a2-b270-73f4ec949841/20170531213326565056.json

@@ -27,7 +27,8 @@
                 "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"
             ],
             "spec_version": "2.1",
-            "type": "malware"
+            "type": "malware",
+            "is_family": false
         }
     ],
     "type": "bundle"

stix2/test/v21/stix2_data/malware/malware--96b08451-b27a-4ff6-893f-790e26393a8e/20170531213248482655.json

@@ -27,7 +27,8 @@
                 "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"
             ],
             "spec_version": "2.1",
-            "type": "malware"
+            "type": "malware",
+            "is_family": false
         }
     ],
     "type": "bundle"

stix2/test/v21/stix2_data/malware/malware--b42378e0-f147-496f-992a-26a49705395b/20170531213215263882.json

@@ -26,7 +26,8 @@
             "object_marking_refs": [
                 "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"
             ],
-            "type": "malware"
+            "type": "malware",
+            "is_family": false
         }
     ],
     "spec_version": "2.0",

stix2/test/v21/test_bundle.py

@@ -4,6 +4,7 @@ import pytest
 
 import stix2
 
+from ...exceptions import InvalidValueError
 from .constants import IDENTITY_ID
 
 EXPECTED_BUNDLE = """{
@@ -20,6 +21,7 @@ EXPECTED_BUNDLE = """{
                 "malicious-activity"
             ],
             "pattern": "[file:hashes.MD5 = 'd41d8cd98f00b204e9800998ecf8427e']",
+            "pattern_type": "stix",
             "valid_from": "2017-01-01T12:34:56Z"
         },
         {
@@ -31,7 +33,8 @@ EXPECTED_BUNDLE = """{
             "name": "Cryptolocker",
             "malware_types": [
                 "ransomware"
-            ]
+            ],
+            "is_family": false
         },
         {
             "type": "relationship",
@@ -57,6 +60,7 @@ EXPECTED_BUNDLE_DICT = {
             "created": "2017-01-01T12:34:56.000Z",
             "modified": "2017-01-01T12:34:56.000Z",
             "pattern": "[file:hashes.MD5 = 'd41d8cd98f00b204e9800998ecf8427e']",
+            "pattern_type": "stix",
             "valid_from": "2017-01-01T12:34:56Z",
             "indicator_types": [
                 "malicious-activity",
@@ -72,6 +76,7 @@ EXPECTED_BUNDLE_DICT = {
             "malware_types": [
                 "ransomware",
             ],
+            "is_family": False,
         },
         {
             "type": "relationship",
@@ -160,15 +165,15 @@ def test_create_bundle_with_arg_listarg_and_kwarg(indicator, malware, relationsh
 
 
 def test_create_bundle_invalid(indicator, malware, relationship):
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.v21.Bundle(objects=[1])
     assert excinfo.value.reason == "This property may only contain a dictionary or object"
 
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.v21.Bundle(objects=[{}])
     assert excinfo.value.reason == "This property may only contain a non-empty dictionary or object"
 
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.v21.Bundle(objects=[{'type': 'bundle'}])
     assert excinfo.value.reason == 'This property may not contain a Bundle object'
 
@@ -179,7 +184,7 @@ def test_parse_bundle(version):
 
     assert bundle.type == "bundle"
     assert bundle.id.startswith("bundle--")
-    assert type(bundle.objects[0]) is stix2.v21.Indicator
+    assert isinstance(bundle.objects[0], stix2.v21.Indicator)
     assert bundle.objects[0].type == 'indicator'
     assert bundle.objects[1].type == 'malware'
     assert bundle.objects[2].type == 'relationship'
@@ -232,6 +237,7 @@ def test_bundle_obj_id_found():
                     "malicious-activity",
                 ],
                 "pattern": "[file:hashes.MD5 = 'd41d8cd98f00b204e9800998ecf8427e']",
+                "pattern_type": "stix",
                 "valid_from": "2017-01-01T12:34:56Z",
             },
             {
@@ -244,6 +250,7 @@ def test_bundle_obj_id_found():
                 "malware_types": [
                     "ransomware",
                 ],
+                "is_family": False,
             },
             {
                 "type": "malware",
@@ -255,6 +262,7 @@ def test_bundle_obj_id_found():
                 "malware_types": [
                     "ransomware",
                 ],
+                "is_family": False,
             },
             {
                 "type": "relationship",

stix2/test/v21/test_core.py

@@ -16,6 +16,7 @@ BUNDLE = {
             "created": "2017-01-01T12:34:56.000Z",
             "modified": "2017-01-01T12:34:56.000Z",
             "pattern": "[file:hashes.MD5 = 'd41d8cd98f00b204e9800998ecf8427e']",
+            "pattern_type": "stix",
             "valid_from": "2017-01-01T12:34:56Z",
             "indicator_types": [
                 "malicious-activity",
@@ -31,6 +32,7 @@ BUNDLE = {
             "malware_types": [
                 "ransomware",
             ],
+            "is_family": False,
         },
         {
             "type": "relationship",
@@ -77,7 +79,7 @@ def test_register_object_with_version():
     v = 'v21'
 
     assert bundle.objects[0].type in core.STIX2_OBJ_MAPS[v]['objects']
-    assert v in str(bundle.objects[0].__class__)
+    assert bundle.objects[0].spec_version == "2.1"
 
 
 def test_register_marking_with_version():
@@ -125,7 +127,7 @@ def test_register_observable_with_default_version():
             "1": {
                 "type": "directory",
                 "path": "/usr/home",
-                "contains_refs": ["0"],
+                "contains_refs": ["file--420bc087-8b53-5ae9-8210-20d27d5e96c8"],
             },
         },
     )
@@ -163,7 +165,7 @@ def test_register_observable_extension_with_default_version():
             "1": {
                 "type": "directory",
                 "path": "/usr/home",
-                "contains_refs": ["0"],
+                "contains_refs": ["file--420bc087-8b53-5ae9-8210-20d27d5e96c8"],
             },
         },
     )

stix2/test/v21/test_course_of_action.py

@@ -1,13 +1,12 @@
-import datetime as dt
+import json
 
 import pytest
-import pytz
 
 import stix2
+import stix2.exceptions
+import stix2.utils
 
-from .constants import COURSE_OF_ACTION_ID, IDENTITY_ID
-
-EXPECTED = """{
+COA_WITH_BIN_JSON = """{
     "type": "course-of-action",
     "spec_version": "2.1",
     "id": "course-of-action--8e2e2d2b-17d4-4cbf-938f-98ee46b3cd3f",
@@ -15,48 +14,85 @@ EXPECTED = """{
     "created": "2016-04-06T20:03:48.000Z",
     "modified": "2016-04-06T20:03:48.000Z",
     "name": "Add TCP port 80 Filter Rule to the existing Block UDP 1434 Filter",
-    "description": "This is how to add a filter rule to block inbound access to TCP port 80 to the existing UDP 1434 filter ..."
+    "description": "This is how to add a filter rule to block inbound access to TCP port 80 to the existing UDP 1434 filter ...",
+    "action_type": "textual:text/plain",
+    "os_execution_envs": [
+        "a",
+        "b",
+        "c"
+    ],
+    "action_bin": "aGVsbG8gd29ybGQ="
 }"""
 
 
-def test_course_of_action_example():
-    coa = stix2.v21.CourseOfAction(
-        id=COURSE_OF_ACTION_ID,
-        created_by_ref=IDENTITY_ID,
-        created="2016-04-06T20:03:48.000Z",
-        modified="2016-04-06T20:03:48.000Z",
-        name="Add TCP port 80 Filter Rule to the existing Block UDP 1434 Filter",
-        description="This is how to add a filter rule to block inbound access to TCP port 80 to the existing UDP 1434 filter ...",
-    )
+COA_WITH_REF_JSON = """{
+    "type": "course-of-action",
+    "spec_version": "2.1",
+    "id": "course-of-action--8e2e2d2b-17d4-4cbf-938f-98ee46b3cd3f",
+    "created_by_ref": "identity--311b2d2d-f010-4473-83ec-1edf84858f4c",
+    "created": "2016-04-06T20:03:48.000Z",
+    "modified": "2016-04-06T20:03:48.000Z",
+    "name": "Add TCP port 80 Filter Rule to the existing Block UDP 1434 Filter",
+    "description": "This is how to add a filter rule to block inbound access to TCP port 80 to the existing UDP 1434 filter ...",
+    "action_type": "textual:text/plain",
+    "os_execution_envs": [
+        "a",
+        "b",
+        "c"
+    ],
+    "action_reference": {
+        "source_name": "a source",
+        "description": "description of a source"
+    }
+}"""
 
-    assert str(coa) == EXPECTED
+
+COA_WITH_BIN_DICT = json.loads(COA_WITH_BIN_JSON)
+COA_WITH_REF_DICT = json.loads(COA_WITH_REF_JSON)
 
 
 @pytest.mark.parametrize(
-    "data", [
-        EXPECTED,
-        {
-            "created": "2016-04-06T20:03:48.000Z",
-            "created_by_ref": IDENTITY_ID,
-            "description": "This is how to add a filter rule to block inbound access to TCP port 80 to the existing UDP 1434 filter ...",
-            "id": COURSE_OF_ACTION_ID,
-            "modified": "2016-04-06T20:03:48.000Z",
-            "name": "Add TCP port 80 Filter Rule to the existing Block UDP 1434 Filter",
-            "spec_version": "2.1",
-            "type": "course-of-action",
-        },
+    "sdo_json,sdo_dict", [
+        (COA_WITH_BIN_JSON, COA_WITH_BIN_DICT),
+        (COA_WITH_REF_JSON, COA_WITH_REF_DICT),
     ],
 )
-def test_parse_course_of_action(data):
-    coa = stix2.parse(data, version="2.1")
+def test_course_of_action_example(sdo_json, sdo_dict):
+    coa = stix2.v21.CourseOfAction(**sdo_dict)
+    assert str(coa) == sdo_json
 
-    assert coa.type == 'course-of-action'
-    assert coa.spec_version == '2.1'
-    assert coa.id == COURSE_OF_ACTION_ID
-    assert coa.created == dt.datetime(2016, 4, 6, 20, 3, 48, tzinfo=pytz.utc)
-    assert coa.modified == dt.datetime(2016, 4, 6, 20, 3, 48, tzinfo=pytz.utc)
-    assert coa.created_by_ref == IDENTITY_ID
-    assert coa.description == "This is how to add a filter rule to block inbound access to TCP port 80 to the existing UDP 1434 filter ..."
-    assert coa.name == "Add TCP port 80 Filter Rule to the existing Block UDP 1434 Filter"
+
+@pytest.mark.parametrize(
+    "sdo_json,sdo_dict", [
+        (COA_WITH_BIN_JSON, COA_WITH_BIN_DICT),
+        (COA_WITH_REF_JSON, COA_WITH_REF_DICT),
+    ],
+)
+def test_parse_course_of_action(sdo_json, sdo_dict):
+
+    # Names of timestamp-valued attributes
+    ts_attrs = {"created", "modified"}
+
+    for data in (sdo_json, sdo_dict):
+        coa = stix2.parse(data, version="2.1")
+
+        # sdo_dict is handy as a source of attribute names/values to check
+        for attr_name, attr_value in sdo_dict.items():
+            cmp_value = stix2.utils.parse_into_datetime(attr_value) \
+                if attr_name in ts_attrs else attr_value
+
+            assert getattr(coa, attr_name) == cmp_value
+
+
+def test_course_of_action_constraint():
+    with pytest.raises(stix2.exceptions.MutuallyExclusivePropertiesError):
+        stix2.v21.CourseOfAction(
+            name="Add TCP port 80 Filter Rule to the existing Block UDP 1434 Filter",
+            action_bin="aGVsbG8gd29ybGQ=",
+            action_reference=stix2.v21.ExternalReference(
+                source_name="a source",
+                description="description of a source",
+            ),
+        )
 
 # TODO: Add other examples

stix2/test/v21/test_custom.py

@@ -2,7 +2,9 @@ import pytest
 
 import stix2
 import stix2.base
+import stix2.v21
 
+from ...exceptions import InvalidValueError
 from .constants import FAKE_TIME, IDENTITY_ID, MARKING_DEFINITION_ID
 
 IDENTITY_CUSTOM_PROP = stix2.v21.Identity(
@@ -97,7 +99,7 @@ def test_identity_custom_property_allowed():
 def test_parse_identity_custom_property(data):
     with pytest.raises(stix2.exceptions.ExtraPropertiesError) as excinfo:
         stix2.parse(data, version="2.1")
-    assert excinfo.value.cls == stix2.v21.Identity
+    assert issubclass(excinfo.value.cls, stix2.v21.Identity)
     assert excinfo.value.properties == ['foo']
     assert "Unexpected properties for" in str(excinfo.value)
 
@@ -136,7 +138,7 @@ def test_custom_property_dict_in_bundled_object():
         'identity_class': 'individual',
         'x_foo': 'bar',
     }
-    with pytest.raises(stix2.exceptions.ExtraPropertiesError):
+    with pytest.raises(InvalidValueError):
         stix2.v21.Bundle(custom_identity)
 
     bundle = stix2.v21.Bundle(custom_identity, allow_custom=True)
@@ -203,7 +205,7 @@ def test_custom_property_object_in_observable_extension():
 
 
 def test_custom_property_dict_in_observable_extension():
-    with pytest.raises(stix2.exceptions.ExtraPropertiesError):
+    with pytest.raises(InvalidValueError):
         stix2.v21.File(
             name='test',
             extensions={
@@ -722,7 +724,7 @@ def test_custom_extension():
 def test_custom_extension_wrong_observable_type():
     # NewExtension is an extension of DomainName, not File
     ext = NewExtension(property1='something')
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.v21.File(
             name="abc.txt",
             extensions={
@@ -888,6 +890,49 @@ def test_parse_observable_with_custom_extension():
     assert parsed.extensions['x-new-ext'].property2 == 12
 
 
+def test_custom_and_spec_extension_mix():
+    """
+    Try to make sure that when allow_custom=True, encountering a custom
+    extension doesn't result in a completely uncleaned extensions property.
+    """
+
+    file_obs = stix2.v21.File(
+        name="my_file.dat",
+        extensions={
+            "x-custom1": {
+                "a": 1,
+                "b": 2,
+            },
+            "ntfs-ext": {
+                "sid": "S-1-whatever",
+            },
+            "x-custom2": {
+                "z": 99.9,
+                "y": False,
+            },
+            "raster-image-ext": {
+                "image_height": 1024,
+                "image_width": 768,
+                "bits_per_pixel": 32,
+            },
+        },
+        allow_custom=True,
+    )
+
+    assert file_obs.extensions["x-custom1"] == {"a": 1, "b": 2}
+    assert file_obs.extensions["x-custom2"] == {"y": False, "z": 99.9}
+    assert file_obs.extensions["ntfs-ext"].sid == "S-1-whatever"
+    assert file_obs.extensions["raster-image-ext"].image_height == 1024
+
+    # Both of these should have been converted to objects, not left as dicts.
+    assert isinstance(
+        file_obs.extensions["raster-image-ext"], stix2.v21.RasterImageExt,
+    )
+    assert isinstance(
+        file_obs.extensions["ntfs-ext"], stix2.v21.NTFSExt,
+    )
+
+
 @pytest.mark.parametrize(
     "data", [
         # URL is not in EXT_MAP
@@ -915,7 +960,7 @@ def test_parse_observable_with_custom_extension():
     ],
 )
 def test_parse_observable_with_unregistered_custom_extension(data):
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.parse_observable(data, version='2.1')
     assert "Can't parse unknown extension type" in str(excinfo.value)
 

stix2/test/v21/test_datastore_filters.py

@@ -16,6 +16,7 @@ stix_objs = [
             "remote-access-trojan",
         ],
         "modified": "2017-01-27T13:49:53.997Z",
+        "is_family": False,
         "name": "Poison Ivy",
         "type": "malware",
     },
@@ -28,6 +29,7 @@ stix_objs = [
         "modified": "2014-05-08T09:00:00.000Z",
         "name": "File hash for Poison Ivy variant",
         "pattern": "[file:hashes.'SHA-256' = 'ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c']",
+        "pattern_type": "stix",
         "spec_version": "2.1",
         "type": "indicator",
         "valid_from": "2014-05-08T09:00:00.000000Z",
@@ -84,7 +86,10 @@ stix_objs = [
         "objects": {
             "0": {
                 "type": "file",
+                "spec_version": "2.1",
+                "id": "file--42a7175a-42cc-508f-8fa7-23b330aff876",
                 "name": "HAL 9000.exe",
+                "defanged": False,
             },
         },
 
@@ -107,8 +112,14 @@ filters = [
     Filter("object_marking_refs", "=", "marking-definition--613f2e26-0000-4000-8000-b8e91df99dc9"),
     Filter("granular_markings.selectors", "in", "description"),
     Filter("external_references.source_name", "=", "CVE"),
-    Filter("objects", "=", {"0": {"type": "file", "name": "HAL 9000.exe"}}),
-    Filter("objects", "contains", {"type": "file", "name": "HAL 9000.exe"}),
+    Filter(
+        "objects", "=",
+        {"0": {"type": "file", "id": "file--42a7175a-42cc-508f-8fa7-23b330aff876", "name": "HAL 9000.exe", "spec_version": "2.1", "defanged": False}},
+    ),
+    Filter(
+        "objects", "contains",
+        {"type": "file", "id": "file--42a7175a-42cc-508f-8fa7-23b330aff876", "name": "HAL 9000.exe", "spec_version": "2.1", "defanged": False},
+    ),
     Filter("labels", "contains", "heartbleed"),
 ]
 

stix2/test/v21/test_datastore_memory.py

@@ -24,6 +24,7 @@ IND1 = {
     "modified": "2017-01-27T13:49:53.935Z",
     "name": "Malicious site hosting downloader",
     "pattern": "[url:value = 'http://x4z9arb.cn/4712']",
+    "pattern_type": "stix",
     "spec_version": "2.1",
     "type": "indicator",
     "valid_from": "2017-01-27T13:49:53.935382Z",
@@ -37,6 +38,7 @@ IND2 = {
     "modified": "2017-01-27T13:49:53.935Z",
     "name": "Malicious site hosting downloader",
     "pattern": "[url:value = 'http://x4z9arb.cn/4712']",
+    "pattern_type": "stix",
     "spec_version": "2.1",
     "type": "indicator",
     "valid_from": "2017-01-27T13:49:53.935382Z",
@@ -50,6 +52,7 @@ IND3 = {
     "modified": "2017-01-27T13:49:53.936Z",
     "name": "Malicious site hosting downloader",
     "pattern": "[url:value = 'http://x4z9arb.cn/4712']",
+    "pattern_type": "stix",
     "spec_version": "2.1",
     "type": "indicator",
     "valid_from": "2017-01-27T13:49:53.935382Z",
@@ -63,6 +66,7 @@ IND4 = {
     "modified": "2017-01-27T13:49:53.935Z",
     "name": "Malicious site hosting downloader",
     "pattern": "[url:value = 'http://x4z9arb.cn/4712']",
+    "pattern_type": "stix",
     "spec_version": "2.1",
     "type": "indicator",
     "valid_from": "2017-01-27T13:49:53.935382Z",
@@ -76,6 +80,7 @@ IND5 = {
     "modified": "2017-01-27T13:49:53.935Z",
     "name": "Malicious site hosting downloader",
     "pattern": "[url:value = 'http://x4z9arb.cn/4712']",
+    "pattern_type": "stix",
     "spec_version": "2.1",
     "type": "indicator",
     "valid_from": "2017-01-27T13:49:53.935382Z",
@@ -89,6 +94,7 @@ IND6 = {
     "modified": "2017-01-31T13:49:53.935Z",
     "name": "Malicious site hosting downloader",
     "pattern": "[url:value = 'http://x4z9arb.cn/4712']",
+    "pattern_type": "stix",
     "spec_version": "2.1",
     "type": "indicator",
     "valid_from": "2017-01-27T13:49:53.935382Z",
@@ -102,6 +108,7 @@ IND7 = {
     "modified": "2017-01-27T13:49:53.935Z",
     "name": "Malicious site hosting downloader",
     "pattern": "[url:value = 'http://x4z9arb.cn/4712']",
+    "pattern_type": "stix",
     "spec_version": "2.1",
     "type": "indicator",
     "valid_from": "2017-01-27T13:49:53.935382Z",
@@ -115,6 +122,7 @@ IND8 = {
     "modified": "2017-01-27T13:49:53.935Z",
     "name": "Malicious site hosting downloader",
     "pattern": "[url:value = 'http://x4z9arb.cn/4712']",
+    "pattern_type": "stix",
     "spec_version": "2.1",
     "type": "indicator",
     "valid_from": "2017-01-27T13:49:53.935382Z",

stix2/test/v21/test_environment.py

@@ -1,11 +1,17 @@
 import pytest
 
 import stix2
+import stix2.environment
+import stix2.exceptions
 
 from .constants import (
-    CAMPAIGN_ID, CAMPAIGN_KWARGS, FAKE_TIME, IDENTITY_ID, IDENTITY_KWARGS,
-    INDICATOR_ID, INDICATOR_KWARGS, MALWARE_ID, MALWARE_KWARGS,
-    RELATIONSHIP_IDS,
+    ATTACK_PATTERN_ID, ATTACK_PATTERN_KWARGS, CAMPAIGN_ID, CAMPAIGN_KWARGS,
+    COURSE_OF_ACTION_ID, COURSE_OF_ACTION_KWARGS, FAKE_TIME, IDENTITY_ID,
+    IDENTITY_KWARGS, INDICATOR_ID, INDICATOR_KWARGS, INTRUSION_SET_ID,
+    INTRUSION_SET_KWARGS, LOCATION_ID, MALWARE_ID, MALWARE_KWARGS,
+    OBSERVED_DATA_ID, OBSERVED_DATA_KWARGS, RELATIONSHIP_IDS, REPORT_ID,
+    REPORT_KWARGS, THREAT_ACTOR_ID, THREAT_ACTOR_KWARGS, TOOL_ID, TOOL_KWARGS,
+    VULNERABILITY_ID, VULNERABILITY_KWARGS,
 )
 
 
@@ -219,7 +225,8 @@ def test_parse_malware():
         "name": "Cryptolocker",
         "malware_types": [
             "ransomware"
-        ]
+        ],
+        "is_family": false
     }"""
     mal = env.parse(data, version="2.1")
 
@@ -230,6 +237,7 @@ def test_parse_malware():
     assert mal.modified == FAKE_TIME
     assert mal.malware_types == ['ransomware']
     assert mal.name == "Cryptolocker"
+    assert not mal.is_family
 
 
 def test_creator_of():
@@ -351,6 +359,7 @@ def test_related_to_no_id(ds):
     mal = {
         "type": "malware",
         "name": "some variant",
+        "is_family": False,
     }
     with pytest.raises(ValueError) as excinfo:
         env.related_to(mal)
@@ -372,3 +381,399 @@ def test_related_to_by_target(ds):
     assert len(resp) == 2
     assert any(x['id'] == CAMPAIGN_ID for x in resp)
     assert any(x['id'] == INDICATOR_ID for x in resp)
+
+
+def test_semantic_equivalence_on_same_attack_pattern1():
+    ap1 = stix2.v21.AttackPattern(id=ATTACK_PATTERN_ID, **ATTACK_PATTERN_KWARGS)
+    ap2 = stix2.v21.AttackPattern(id=ATTACK_PATTERN_ID, **ATTACK_PATTERN_KWARGS)
+    env = stix2.Environment().semantically_equivalent(ap1, ap2)
+    assert round(env) == 100
+
+
+def test_semantic_equivalence_on_same_attack_pattern2():
+    ATTACK_KWARGS = dict(
+        name="Phishing",
+        external_references=[
+            {
+                "url": "https://example2",
+                "source_name": "some-source2",
+            },
+        ],
+    )
+    ap1 = stix2.v21.AttackPattern(id=ATTACK_PATTERN_ID, **ATTACK_KWARGS)
+    ap2 = stix2.v21.AttackPattern(id=ATTACK_PATTERN_ID, **ATTACK_KWARGS)
+    env = stix2.Environment().semantically_equivalent(ap1, ap2)
+    assert round(env) == 100
+
+
+def test_semantic_equivalence_on_same_campaign1():
+    camp1 = stix2.v21.Campaign(id=CAMPAIGN_ID, **CAMPAIGN_KWARGS)
+    camp2 = stix2.v21.Campaign(id=CAMPAIGN_ID, **CAMPAIGN_KWARGS)
+    env = stix2.Environment().semantically_equivalent(camp1, camp2)
+    assert round(env) == 100
+
+
+def test_semantic_equivalence_on_same_campaign2():
+    CAMP_KWARGS = dict(
+        name="Green Group Attacks Against Finance",
+        description="Campaign by Green Group against a series of targets in the financial services sector.",
+        aliases=["super-green", "some-green"],
+    )
+    camp1 = stix2.v21.Campaign(id=CAMPAIGN_ID, **CAMP_KWARGS)
+    camp2 = stix2.v21.Campaign(id=CAMPAIGN_ID, **CAMP_KWARGS)
+    env = stix2.Environment().semantically_equivalent(camp1, camp2)
+    assert round(env) == 100
+
+
+def test_semantic_equivalence_on_same_identity1():
+    iden1 = stix2.v21.Identity(id=IDENTITY_ID, **IDENTITY_KWARGS)
+    iden2 = stix2.v21.Identity(id=IDENTITY_ID, **IDENTITY_KWARGS)
+    env = stix2.Environment().semantically_equivalent(iden1, iden2)
+    assert round(env) == 100
+
+
+def test_semantic_equivalence_on_same_identity2():
+    IDEN_KWARGS = dict(
+        name="John Smith",
+        identity_class="individual",
+        sectors=["government", "critical-infrastructure"],
+    )
+    iden1 = stix2.v21.Identity(id=IDENTITY_ID, **IDEN_KWARGS)
+    iden2 = stix2.v21.Identity(id=IDENTITY_ID, **IDEN_KWARGS)
+    env = stix2.Environment().semantically_equivalent(iden1, iden2)
+    assert round(env) == 100
+
+
+def test_semantic_equivalence_on_same_indicator():
+    ind1 = stix2.v21.Indicator(id=INDICATOR_ID, **INDICATOR_KWARGS)
+    ind2 = stix2.v21.Indicator(id=INDICATOR_ID, **INDICATOR_KWARGS)
+    env = stix2.Environment().semantically_equivalent(ind1, ind2)
+    assert round(env) == 100
+
+
+def test_semantic_equivalence_on_same_location1():
+    LOCATION_KWARGS = dict(latitude=45, longitude=179)
+    loc1 = stix2.v21.Location(id=LOCATION_ID, **LOCATION_KWARGS)
+    loc2 = stix2.v21.Location(id=LOCATION_ID, **LOCATION_KWARGS)
+    env = stix2.Environment().semantically_equivalent(loc1, loc2)
+    assert round(env) == 100
+
+
+def test_semantic_equivalence_on_same_location2():
+    LOCATION_KWARGS = dict(
+        latitude=38.889,
+        longitude=-77.023,
+        region="northern-america",
+        country="us",
+    )
+    loc1 = stix2.v21.Location(id=LOCATION_ID, **LOCATION_KWARGS)
+    loc2 = stix2.v21.Location(id=LOCATION_ID, **LOCATION_KWARGS)
+    env = stix2.Environment().semantically_equivalent(loc1, loc2)
+    assert round(env) == 100
+
+
+def test_semantic_equivalence_on_same_malware():
+    malw1 = stix2.v21.Malware(id=MALWARE_ID, **MALWARE_KWARGS)
+    malw2 = stix2.v21.Malware(id=MALWARE_ID, **MALWARE_KWARGS)
+    env = stix2.Environment().semantically_equivalent(malw1, malw2)
+    assert round(env) == 100
+
+
+def test_semantic_equivalence_on_same_threat_actor1():
+    ta1 = stix2.v21.ThreatActor(id=THREAT_ACTOR_ID, **THREAT_ACTOR_KWARGS)
+    ta2 = stix2.v21.ThreatActor(id=THREAT_ACTOR_ID, **THREAT_ACTOR_KWARGS)
+    env = stix2.Environment().semantically_equivalent(ta1, ta2)
+    assert round(env) == 100
+
+
+def test_semantic_equivalence_on_same_threat_actor2():
+    THREAT_KWARGS = dict(
+        threat_actor_types=["crime-syndicate"],
+        aliases=["super-evil"],
+        name="Evil Org",
+    )
+    ta1 = stix2.v21.ThreatActor(id=THREAT_ACTOR_ID, **THREAT_KWARGS)
+    ta2 = stix2.v21.ThreatActor(id=THREAT_ACTOR_ID, **THREAT_KWARGS)
+    env = stix2.Environment().semantically_equivalent(ta1, ta2)
+    assert round(env) == 100
+
+
+def test_semantic_equivalence_on_same_tool():
+    tool1 = stix2.v21.Tool(id=TOOL_ID, **TOOL_KWARGS)
+    tool2 = stix2.v21.Tool(id=TOOL_ID, **TOOL_KWARGS)
+    env = stix2.Environment().semantically_equivalent(tool1, tool2)
+    assert round(env) == 100
+
+
+def test_semantic_equivalence_on_same_vulnerability1():
+    vul1 = stix2.v21.Vulnerability(id=VULNERABILITY_ID, **VULNERABILITY_KWARGS)
+    vul2 = stix2.v21.Vulnerability(id=VULNERABILITY_ID, **VULNERABILITY_KWARGS)
+    env = stix2.Environment().semantically_equivalent(vul1, vul2)
+    assert round(env) == 100
+
+
+def test_semantic_equivalence_on_same_vulnerability2():
+    VULN_KWARGS1 = dict(
+        name="Heartbleed",
+        external_references=[
+            {
+                "url": "https://example",
+                "source_name": "some-source",
+            },
+        ],
+    )
+    VULN_KWARGS2 = dict(
+        name="Zot",
+        external_references=[
+            {
+                "url": "https://example2",
+                "source_name": "some-source2",
+            },
+        ],
+    )
+    vul1 = stix2.v21.Vulnerability(id=VULNERABILITY_ID, **VULN_KWARGS1)
+    vul2 = stix2.v21.Vulnerability(id=VULNERABILITY_ID, **VULN_KWARGS2)
+    env = stix2.Environment().semantically_equivalent(vul1, vul2)
+    assert round(env) == 0.0
+
+
+def test_semantic_equivalence_on_unknown_object():
+    CUSTOM_KWARGS1 = dict(
+        type="x-foobar",
+        id="x-foobar--0c7b5b88-8ff7-4a4d-aa9d-feb398cd0061",
+        name="Heartbleed",
+        external_references=[
+            {
+                "url": "https://example",
+                "source_name": "some-source",
+            },
+        ],
+    )
+    CUSTOM_KWARGS2 = dict(
+        type="x-foobar",
+        id="x-foobar--0c7b5b88-8ff7-4a4d-aa9d-feb398cd0061",
+        name="Zot",
+        external_references=[
+            {
+                "url": "https://example2",
+                "source_name": "some-source2",
+            },
+        ],
+    )
+
+    def _x_foobar_checks(obj1, obj2, **weights):
+        matching_score = 0.0
+        sum_weights = 0.0
+        if stix2.environment.check_property_present("external_references", obj1, obj2):
+            w = weights["external_references"]
+            sum_weights += w
+            matching_score += w * stix2.environment.partial_external_reference_based(
+                obj1["external_references"],
+                obj2["external_references"],
+            )
+        if stix2.environment.check_property_present("name", obj1, obj2):
+            w = weights["name"]
+            sum_weights += w
+            matching_score += w * stix2.environment.partial_string_based(obj1["name"], obj2["name"])
+        return matching_score, sum_weights
+
+    weights = {
+        "x-foobar": {
+            "external_references": 40,
+            "name": 60,
+            "method": _x_foobar_checks,
+        },
+        "_internal": {
+            "ignore_spec_version": False,
+        },
+    }
+    cust1 = stix2.parse(CUSTOM_KWARGS1, allow_custom=True)
+    cust2 = stix2.parse(CUSTOM_KWARGS2, allow_custom=True)
+    env = stix2.Environment().semantically_equivalent(cust1, cust2, **weights)
+    assert round(env) == 0
+
+
+def test_semantic_equivalence_different_type_raises():
+    with pytest.raises(ValueError) as excinfo:
+        vul1 = stix2.v21.Vulnerability(id=VULNERABILITY_ID, **VULNERABILITY_KWARGS)
+        ind1 = stix2.v21.Indicator(id=INDICATOR_ID, **INDICATOR_KWARGS)
+        stix2.Environment().semantically_equivalent(vul1, ind1)
+
+    assert str(excinfo.value) == "The objects to compare must be of the same type!"
+
+
+def test_semantic_equivalence_different_spec_version_raises():
+    with pytest.raises(ValueError) as excinfo:
+        V20_KWARGS = dict(
+            labels=['malicious-activity'],
+            pattern="[file:hashes.MD5 = 'd41d8cd98f00b204e9800998ecf8427e']",
+        )
+        ind1 = stix2.v21.Indicator(id=INDICATOR_ID, **INDICATOR_KWARGS)
+        ind2 = stix2.v20.Indicator(id=INDICATOR_ID, **V20_KWARGS)
+        stix2.Environment().semantically_equivalent(ind1, ind2)
+
+    assert str(excinfo.value) == "The objects to compare must be of the same spec version!"
+
+
+@pytest.mark.parametrize(
+    "obj1,obj2,ret_val",
+    [
+        (
+             stix2.v21.CourseOfAction(id=COURSE_OF_ACTION_ID, **COURSE_OF_ACTION_KWARGS),
+             stix2.v21.CourseOfAction(id=COURSE_OF_ACTION_ID, **COURSE_OF_ACTION_KWARGS),
+             "course-of-action type has no semantic equivalence implementation!",
+        ),
+        (
+             stix2.v21.IntrusionSet(id=INTRUSION_SET_ID, **INTRUSION_SET_KWARGS),
+             stix2.v21.IntrusionSet(id=INTRUSION_SET_ID, **INTRUSION_SET_KWARGS),
+             "intrusion-set type has no semantic equivalence implementation!",
+        ),
+        (
+             stix2.v21.ObservedData(id=OBSERVED_DATA_ID, **OBSERVED_DATA_KWARGS),
+             stix2.v21.ObservedData(id=OBSERVED_DATA_ID, **OBSERVED_DATA_KWARGS),
+             "observed-data type has no semantic equivalence implementation!",
+        ),
+        (
+             stix2.v21.Report(id=REPORT_ID, **REPORT_KWARGS),
+             stix2.v21.Report(id=REPORT_ID, **REPORT_KWARGS),
+             "report type has no semantic equivalence implementation!",
+        ),
+    ],
+)
+def test_semantic_equivalence_on_unsupported_types(obj1, obj2, ret_val):
+    with pytest.raises(stix2.exceptions.SemanticEquivalenceUnsupportedTypeError) as excinfo:
+        stix2.Environment().semantically_equivalent(obj1, obj2)
+    assert ret_val == str(excinfo.value)
+
+
+def test_semantic_equivalence_zero_match():
+    IND_KWARGS = dict(
+        indicator_types=["APTX"],
+        pattern="[ipv4-addr:value = '192.168.1.1']",
+        pattern_type="stix",
+        valid_from="2019-01-01T12:34:56Z",
+    )
+    weights = {
+        "indicator": {
+            "indicator_types": 15,
+            "pattern": 80,
+            "valid_from": 0,
+            "tdelta": 1,  # One day interval
+            "method": stix2.environment._indicator_checks,
+        },
+        "_internal": {
+            "ignore_spec_version": False,
+        },
+    }
+    ind1 = stix2.v21.Indicator(id=INDICATOR_ID, **INDICATOR_KWARGS)
+    ind2 = stix2.v21.Indicator(id=INDICATOR_ID, **IND_KWARGS)
+    env = stix2.Environment().semantically_equivalent(ind1, ind2, **weights)
+    assert round(env) == 0
+
+
+def test_semantic_equivalence_different_spec_version():
+    IND_KWARGS = dict(
+        labels=["APTX"],
+        pattern="[ipv4-addr:value = '192.168.1.1']",
+    )
+    weights = {
+        "indicator": {
+            "indicator_types": 15,
+            "pattern": 80,
+            "valid_from": 0,
+            "tdelta": 1,  # One day interval
+            "method": stix2.environment._indicator_checks,
+        },
+        "_internal": {
+            "ignore_spec_version": True,  # Disables spec_version check.
+        },
+    }
+    ind1 = stix2.v21.Indicator(id=INDICATOR_ID, **INDICATOR_KWARGS)
+    ind2 = stix2.v20.Indicator(id=INDICATOR_ID, **IND_KWARGS)
+    env = stix2.Environment().semantically_equivalent(ind1, ind2, **weights)
+    assert round(env) == 0
+
+
+@pytest.mark.parametrize(
+    "refs1,refs2,ret_val", [
+        (
+            [
+                {
+                    "url": "https://attack.mitre.org/techniques/T1150",
+                    "source_name": "mitre-attack",
+                    "external_id": "T1150",
+                },
+                {
+                    "url": "https://researchcenter.paloaltonetworks.com/2016/09/unit42-sofacys-komplex-os-x-trojan/",
+                    "source_name": "Sofacy Komplex Trojan",
+                    "description": "Dani Creus, Tyler Halfpop, Robert Falcone. (2016, September 26). Sofacy's 'Komplex' OS X Trojan. Retrieved July 8, 2017.",
+                },
+            ],
+            [
+                {
+                    "url": "https://attack.mitre.org/techniques/T1129",
+                    "source_name": "mitre-attack",
+                    "external_id": "T1129",
+                },
+                {
+                    "url": "https://en.wikipedia.org/wiki/Microsoft_Windows_library_files",
+                    "source_name": "Wikipedia Windows Library Files",
+                    "description": "Wikipedia. (2017, January 31). Microsoft Windows library files. Retrieved February 13, 2017.",
+                },
+            ],
+            0.0,
+        ),
+        (
+            [
+                {
+                    "url": "https://attack.mitre.org/techniques/T1129",
+                    "source_name": "mitre-attack",
+                    "external_id": "T1129",
+                },
+            ],
+            [
+                {
+                    "url": "https://attack.mitre.org/techniques/T1129",
+                    "source_name": "mitre-attack",
+                    "external_id": "T1129",
+                },
+                {
+                    "url": "https://en.wikipedia.org/wiki/Microsoft_Windows_library_files",
+                    "source_name": "Wikipedia Windows Library Files",
+                    "description": "Wikipedia. (2017, January 31). Microsoft Windows library files. Retrieved February 13, 2017.",
+                },
+            ],
+            1.0,
+        ),
+        (
+            [
+                {
+                    "url": "https://example",
+                    "source_name": "some-source",
+                },
+            ],
+            [
+                {
+                    "url": "https://example",
+                    "source_name": "some-source",
+                },
+            ],
+            1.0,
+        ),
+    ],
+)
+def test_semantic_equivalence_external_references(refs1, refs2, ret_val):
+    value = stix2.environment.partial_external_reference_based(refs1, refs2)
+    assert value == ret_val
+
+
+def test_semantic_equivalence_timetamp():
+    t1 = "2018-10-17T00:14:20.652Z"
+    t2 = "2018-10-17T12:14:20.652Z"
+    assert stix2.environment.partial_timestamp_based(t1, t2, 1) == 0.5
+
+
+def test_semantic_equivalence_exact_match():
+    t1 = "2018-10-17T00:14:20.652Z"
+    t2 = "2018-10-17T12:14:20.652Z"
+    assert stix2.environment.exact_match(t1, t2) == 0.0

stix2/test/v21/test_granular_markings.py

@@ -1,7 +1,7 @@
 import pytest
 
 from stix2 import markings
-from stix2.exceptions import MarkingNotFoundError
+from stix2.exceptions import InvalidSelectorError, MarkingNotFoundError
 from stix2.v21 import TLP_RED, Malware
 
 from .constants import MALWARE_MORE_KWARGS as MALWARE_KWARGS_CONST
@@ -209,7 +209,7 @@ def test_add_marking_mark_same_property_same_marking():
     ],
 )
 def test_add_marking_bad_selector(data, marking):
-    with pytest.raises(AssertionError):
+    with pytest.raises(InvalidSelectorError):
         markings.add_markings(data, marking[0], marking[1])
 
 
@@ -329,7 +329,7 @@ def test_get_markings_multiple_selectors(data):
 )
 def test_get_markings_bad_selector(data, selector):
     """Test bad selectors raise exception"""
-    with pytest.raises(AssertionError):
+    with pytest.raises(InvalidSelectorError):
         markings.get_markings(data, selector)
 
 
@@ -714,7 +714,7 @@ def test_remove_marking_bad_selector():
     before = {
         "description": "test description",
     }
-    with pytest.raises(AssertionError):
+    with pytest.raises(InvalidSelectorError):
         markings.remove_markings(before, ["marking-definition--1", "marking-definition--2"], ["title"])
 
 
@@ -805,7 +805,7 @@ def test_is_marked_smoke(data):
 )
 def test_is_marked_invalid_selector(data, selector):
     """Test invalid selector raises an error."""
-    with pytest.raises(AssertionError):
+    with pytest.raises(InvalidSelectorError):
         markings.is_marked(data, selectors=selector)
 
 
@@ -1000,7 +1000,7 @@ def test_is_marked_positional_arguments_combinations():
 
 
 def test_create_sdo_with_invalid_marking():
-    with pytest.raises(AssertionError) as excinfo:
+    with pytest.raises(InvalidSelectorError) as excinfo:
         Malware(
             granular_markings=[
                 {
@@ -1192,7 +1192,7 @@ def test_set_marking_bad_selector(marking):
         **MALWARE_KWARGS
     )
 
-    with pytest.raises(AssertionError):
+    with pytest.raises(InvalidSelectorError):
         before = markings.set_markings(before, marking[0], marking[1])
 
     assert before == after
@@ -1298,7 +1298,7 @@ def test_clear_marking_all_selectors(data):
 )
 def test_clear_marking_bad_selector(data, selector):
     """Test bad selector raises exception."""
-    with pytest.raises(AssertionError):
+    with pytest.raises(InvalidSelectorError):
         markings.clear_markings(data, selector)
 
 

stix2/test/v21/test_grouping.py

@@ -0,0 +1,128 @@
+import datetime as dt
+
+import pytest
+import pytz
+
+import stix2
+
+from .constants import FAKE_TIME, GROUPING_ID, GROUPING_KWARGS
+
+EXPECTED_GROUPING = """{
+    "type": "grouping",
+    "spec_version": "2.1",
+    "id": "grouping--753abcde-3141-5926-ace5-0a810b1ff996",
+    "created": "2017-01-01T12:34:56.000Z",
+    "modified": "2017-01-01T12:34:56.000Z",
+    "name": "Harry Potter and the Leet Hackers",
+    "context": "suspicious-activity",
+    "object_refs": [
+        "malware--c8d2fae5-7271-400c-b81d-931a4caf20b9",
+        "identity--988145ed-a3b4-4421-b7a7-273376be67ce"
+    ]
+}"""
+
+
+def test_grouping_with_all_required_properties():
+    now = dt.datetime(2017, 1, 1, 12, 34, 56, tzinfo=pytz.utc)
+
+    grp = stix2.v21.Grouping(
+        type="grouping",
+        id=GROUPING_ID,
+        created=now,
+        modified=now,
+        name="Harry Potter and the Leet Hackers",
+        context="suspicious-activity",
+        object_refs=[
+            "malware--c8d2fae5-7271-400c-b81d-931a4caf20b9",
+            "identity--988145ed-a3b4-4421-b7a7-273376be67ce",
+        ],
+    )
+
+    assert str(grp) == EXPECTED_GROUPING
+
+
+def test_grouping_autogenerated_properties(grouping):
+    assert grouping.type == 'grouping'
+    assert grouping.id == 'grouping--00000000-0000-4000-8000-000000000001'
+    assert grouping.created == FAKE_TIME
+    assert grouping.modified == FAKE_TIME
+    assert grouping.name == "Harry Potter and the Leet Hackers"
+    assert grouping.context == "suspicious-activity"
+
+    assert grouping['type'] == 'grouping'
+    assert grouping['id'] == 'grouping--00000000-0000-4000-8000-000000000001'
+    assert grouping['created'] == FAKE_TIME
+    assert grouping['modified'] == FAKE_TIME
+    assert grouping['name'] == "Harry Potter and the Leet Hackers"
+    assert grouping['context'] == "suspicious-activity"
+
+
+def test_grouping_type_must_be_grouping():
+    with pytest.raises(stix2.exceptions.InvalidValueError) as excinfo:
+        stix2.v21.Grouping(type='xxx', **GROUPING_KWARGS)
+
+    assert excinfo.value.cls == stix2.v21.Grouping
+    assert excinfo.value.prop_name == "type"
+    assert excinfo.value.reason == "must equal 'grouping'."
+    assert str(excinfo.value) == "Invalid value for Grouping 'type': must equal 'grouping'."
+
+
+def test_grouping_id_must_start_with_grouping():
+    with pytest.raises(stix2.exceptions.InvalidValueError) as excinfo:
+        stix2.v21.Grouping(id='my-prefix--', **GROUPING_KWARGS)
+
+    assert excinfo.value.cls == stix2.v21.Grouping
+    assert excinfo.value.prop_name == "id"
+    assert excinfo.value.reason == "must start with 'grouping--'."
+    assert str(excinfo.value) == "Invalid value for Grouping 'id': must start with 'grouping--'."
+
+
+def test_grouping_required_properties():
+    with pytest.raises(stix2.exceptions.MissingPropertiesError) as excinfo:
+        stix2.v21.Grouping()
+
+    assert excinfo.value.cls == stix2.v21.Grouping
+    assert excinfo.value.properties == ["context", "object_refs"]
+
+
+def test_invalid_kwarg_to_grouping():
+    with pytest.raises(stix2.exceptions.ExtraPropertiesError) as excinfo:
+        stix2.v21.Grouping(my_custom_property="foo", **GROUPING_KWARGS)
+
+    assert excinfo.value.cls == stix2.v21.Grouping
+    assert excinfo.value.properties == ['my_custom_property']
+    assert str(excinfo.value) == "Unexpected properties for Grouping: (my_custom_property)."
+
+
+@pytest.mark.parametrize(
+    "data", [
+        EXPECTED_GROUPING,
+        {
+            "type": "grouping",
+            "spec_version": "2.1",
+            "id": GROUPING_ID,
+            "created": "2017-01-01T12:34:56.000Z",
+            "modified": "2017-01-01T12:34:56.000Z",
+            "name": "Harry Potter and the Leet Hackers",
+            "context": "suspicious-activity",
+            "object_refs": [
+                "malware--c8d2fae5-7271-400c-b81d-931a4caf20b9",
+                "identity--988145ed-a3b4-4421-b7a7-273376be67ce",
+            ],
+        },
+    ],
+)
+def test_parse_grouping(data):
+    grp = stix2.parse(data)
+
+    assert grp.type == 'grouping'
+    assert grp.spec_version == '2.1'
+    assert grp.id == GROUPING_ID
+    assert grp.created == dt.datetime(2017, 1, 1, 12, 34, 56, tzinfo=pytz.utc)
+    assert grp.modified == dt.datetime(2017, 1, 1, 12, 34, 56, tzinfo=pytz.utc)
+    assert grp.name == "Harry Potter and the Leet Hackers"
+    assert grp.context == "suspicious-activity"
+    assert grp.object_refs == [
+        "malware--c8d2fae5-7271-400c-b81d-931a4caf20b9",
+        "identity--988145ed-a3b4-4421-b7a7-273376be67ce",
+    ]

stix2/test/v21/test_indicator.py

@@ -18,6 +18,7 @@ EXPECTED_INDICATOR = """{
         "malicious-activity"
     ],
     "pattern": "[file:hashes.MD5 = 'd41d8cd98f00b204e9800998ecf8427e']",
+    "pattern_type": "stix",
     "valid_from": "1970-01-01T00:00:01Z"
 }"""
 
@@ -29,6 +30,7 @@ EXPECTED_INDICATOR_REPR = "Indicator(" + " ".join("""
     modified='2017-01-01T00:00:01.000Z',
     indicator_types=['malicious-activity'],
     pattern="[file:hashes.MD5 = 'd41d8cd98f00b204e9800998ecf8427e']",
+    pattern_type='stix',
     valid_from='1970-01-01T00:00:01Z'
 """.split()) + ")"
 
@@ -43,6 +45,7 @@ def test_indicator_with_all_required_properties():
         created=now,
         modified=now,
         pattern="[file:hashes.MD5 = 'd41d8cd98f00b204e9800998ecf8427e']",
+        pattern_type="stix",
         valid_from=epoch,
         indicator_types=['malicious-activity'],
     )
@@ -98,8 +101,8 @@ def test_indicator_required_properties():
         stix2.v21.Indicator()
 
     assert excinfo.value.cls == stix2.v21.Indicator
-    assert excinfo.value.properties == ["indicator_types", "pattern"]
-    assert str(excinfo.value) == "No values for required properties for Indicator: (indicator_types, pattern)."
+    assert excinfo.value.properties == ["indicator_types", "pattern", "pattern_type", "valid_from"]
+    assert str(excinfo.value) == "No values for required properties for Indicator: (indicator_types, pattern, pattern_type, valid_from)."
 
 
 def test_indicator_required_property_pattern():
@@ -107,7 +110,7 @@ def test_indicator_required_property_pattern():
         stix2.v21.Indicator(indicator_types=['malicious-activity'])
 
     assert excinfo.value.cls == stix2.v21.Indicator
-    assert excinfo.value.properties == ["pattern"]
+    assert excinfo.value.properties == ["pattern", "pattern_type", "valid_from"]
 
 
 def test_indicator_created_ref_invalid_format():
@@ -116,8 +119,6 @@ def test_indicator_created_ref_invalid_format():
 
     assert excinfo.value.cls == stix2.v21.Indicator
     assert excinfo.value.prop_name == "created_by_ref"
-    assert excinfo.value.reason == "must start with 'identity'."
-    assert str(excinfo.value) == "Invalid value for Indicator 'created_by_ref': must start with 'identity'."
 
 
 def test_indicator_revoked_invalid():
@@ -164,6 +165,7 @@ def test_created_modified_time_are_identical_by_default():
                 "malicious-activity",
             ],
             "pattern": "[file:hashes.MD5 = 'd41d8cd98f00b204e9800998ecf8427e']",
+            "pattern_type": "stix",
             "valid_from": "1970-01-01T00:00:01Z",
         },
     ],
@@ -186,6 +188,8 @@ def test_invalid_indicator_pattern():
         stix2.v21.Indicator(
             indicator_types=['malicious-activity'],
             pattern="file:hashes.MD5 = 'd41d8cd98f00b204e9800998ecf8427e'",
+            pattern_type="stix",
+            valid_from="2017-01-01T12:34:56Z",
         )
     assert excinfo.value.cls == stix2.v21.Indicator
     assert excinfo.value.prop_name == 'pattern'
@@ -195,6 +199,8 @@ def test_invalid_indicator_pattern():
         stix2.v21.Indicator(
             indicator_types=['malicious-activity'],
             pattern='[file:hashes.MD5 = "d41d8cd98f00b204e9800998ecf8427e"]',
+            pattern_type="stix",
+            valid_from="2017-01-01T12:34:56Z",
         )
     assert excinfo.value.cls == stix2.v21.Indicator
     assert excinfo.value.prop_name == 'pattern'

stix2/test/v21/test_infrastructure.py

@@ -0,0 +1,158 @@
+import datetime as dt
+
+import pytest
+import pytz
+
+import stix2
+
+from .constants import FAKE_TIME, INFRASTRUCTURE_ID, INFRASTRUCTURE_KWARGS
+
+EXPECTED_INFRASTRUCTURE = """{
+    "type": "infrastructure",
+    "spec_version": "2.1",
+    "id": "infrastructure--3000ae1b-784c-f03d-8abc-0a625b2ff018",
+    "created": "2017-01-01T12:34:56.000Z",
+    "modified": "2017-01-01T12:34:56.000Z",
+    "name": "Poison Ivy C2",
+    "infrastructure_types": [
+        "command-and-control"
+    ]
+}"""
+
+
+def test_infrastructure_with_all_required_properties():
+    now = dt.datetime(2017, 1, 1, 12, 34, 56, tzinfo=pytz.utc)
+
+    infra = stix2.v21.Infrastructure(
+        type="infrastructure",
+        id=INFRASTRUCTURE_ID,
+        created=now,
+        modified=now,
+        name="Poison Ivy C2",
+        infrastructure_types=["command-and-control"],
+    )
+
+    assert str(infra) == EXPECTED_INFRASTRUCTURE
+
+
+def test_infrastructure_autogenerated_properties(infrastructure):
+    assert infrastructure.type == 'infrastructure'
+    assert infrastructure.id == 'infrastructure--00000000-0000-4000-8000-000000000001'
+    assert infrastructure.created == FAKE_TIME
+    assert infrastructure.modified == FAKE_TIME
+    assert infrastructure.infrastructure_types == ['command-and-control']
+    assert infrastructure.name == "Poison Ivy C2"
+
+    assert infrastructure['type'] == 'infrastructure'
+    assert infrastructure['id'] == 'infrastructure--00000000-0000-4000-8000-000000000001'
+    assert infrastructure['created'] == FAKE_TIME
+    assert infrastructure['modified'] == FAKE_TIME
+    assert infrastructure['infrastructure_types'] == ['command-and-control']
+    assert infrastructure['name'] == "Poison Ivy C2"
+
+
+def test_infrastructure_type_must_be_infrastructure():
+    with pytest.raises(stix2.exceptions.InvalidValueError) as excinfo:
+        stix2.v21.Infrastructure(type='xxx', **INFRASTRUCTURE_KWARGS)
+
+    assert excinfo.value.cls == stix2.v21.Infrastructure
+    assert excinfo.value.prop_name == "type"
+    assert excinfo.value.reason == "must equal 'infrastructure'."
+    assert str(excinfo.value) == "Invalid value for Infrastructure 'type': must equal 'infrastructure'."
+
+
+def test_infrastructure_id_must_start_with_infrastructure():
+    with pytest.raises(stix2.exceptions.InvalidValueError) as excinfo:
+        stix2.v21.Infrastructure(id='my-prefix--', **INFRASTRUCTURE_KWARGS)
+
+    assert excinfo.value.cls == stix2.v21.Infrastructure
+    assert excinfo.value.prop_name == "id"
+    assert excinfo.value.reason == "must start with 'infrastructure--'."
+    assert str(excinfo.value) == "Invalid value for Infrastructure 'id': must start with 'infrastructure--'."
+
+
+def test_infrastructure_required_properties():
+    with pytest.raises(stix2.exceptions.MissingPropertiesError) as excinfo:
+        stix2.v21.Infrastructure()
+
+    assert excinfo.value.cls == stix2.v21.Infrastructure
+    assert excinfo.value.properties == ["infrastructure_types", "name"]
+
+
+def test_infrastructure_required_property_name():
+    with pytest.raises(stix2.exceptions.MissingPropertiesError) as excinfo:
+        stix2.v21.Infrastructure(infrastructure_types=['command-and-control'])
+
+    assert excinfo.value.cls == stix2.v21.Infrastructure
+    assert excinfo.value.properties == ["name"]
+
+
+def test_invalid_kwarg_to_infrastructure():
+    with pytest.raises(stix2.exceptions.ExtraPropertiesError) as excinfo:
+        stix2.v21.Infrastructure(my_custom_property="foo", **INFRASTRUCTURE_KWARGS)
+
+    assert excinfo.value.cls == stix2.v21.Infrastructure
+    assert excinfo.value.properties == ['my_custom_property']
+    assert str(excinfo.value) == "Unexpected properties for Infrastructure: (my_custom_property)."
+
+
+@pytest.mark.parametrize(
+    "data", [
+        EXPECTED_INFRASTRUCTURE,
+        {
+            "type": "infrastructure",
+            "spec_version": "2.1",
+            "id": INFRASTRUCTURE_ID,
+            "created": "2017-01-01T12:34:56.000Z",
+            "modified": "2017-01-01T12:34:56.000Z",
+            "infrastructure_types": ["command-and-control"],
+            "name": "Poison Ivy C2",
+        },
+    ],
+)
+def test_parse_infrastructure(data):
+    infra = stix2.parse(data)
+
+    assert infra.type == 'infrastructure'
+    assert infra.spec_version == '2.1'
+    assert infra.id == INFRASTRUCTURE_ID
+    assert infra.created == dt.datetime(2017, 1, 1, 12, 34, 56, tzinfo=pytz.utc)
+    assert infra.modified == dt.datetime(2017, 1, 1, 12, 34, 56, tzinfo=pytz.utc)
+    assert infra.infrastructure_types == ['command-and-control']
+    assert infra.name == 'Poison Ivy C2'
+
+
+def test_parse_infrastructure_kill_chain_phases():
+    kill_chain = """
+      "kill_chain_phases": [
+        {
+          "kill_chain_name": "lockheed-martin-cyber-kill-chain",
+          "phase_name": "reconnaissance"
+        }
+      ]"""
+    data = EXPECTED_INFRASTRUCTURE.replace('infrastructure"', 'infrastructure",%s' % kill_chain)
+    infra = stix2.parse(data, version="2.1")
+    assert infra.kill_chain_phases[0].kill_chain_name == "lockheed-martin-cyber-kill-chain"
+    assert infra.kill_chain_phases[0].phase_name == "reconnaissance"
+    assert infra['kill_chain_phases'][0]['kill_chain_name'] == "lockheed-martin-cyber-kill-chain"
+    assert infra['kill_chain_phases'][0]['phase_name'] == "reconnaissance"
+
+
+def test_parse_infrastructure_clean_kill_chain_phases():
+    kill_chain = """
+      "kill_chain_phases": [
+        {
+          "kill_chain_name": "lockheed-martin-cyber-kill-chain",
+          "phase_name": 1
+        }
+      ]"""
+    data = EXPECTED_INFRASTRUCTURE.replace('2.1"', '2.1",%s' % kill_chain)
+    infra = stix2.parse(data, version="2.1")
+    assert infra['kill_chain_phases'][0]['phase_name'] == "1"
+
+
+def test_infrastructure_invalid_last_before_first():
+    with pytest.raises(ValueError) as excinfo:
+        stix2.v21.Infrastructure(first_seen="2017-01-01T12:34:56.000Z", last_seen="2017-01-01T12:33:56.000Z", **INFRASTRUCTURE_KWARGS)
+
+    assert "'last_seen' must be greater than or equal to 'first_seen'" in str(excinfo.value)

stix2/test/v21/test_language_content.py

@@ -71,3 +71,18 @@ def test_language_content_campaign():
     # or https://docs.python.org/2/library/json.html#json.dumps
     assert lc.serialize(pretty=True, ensure_ascii=False) == TEST_LANGUAGE_CONTENT
     assert lc.modified == camp.modified
+
+
+def test_object_modified_optional():
+    """
+    object_modified is now optional in STIX 2.1.
+    """
+
+    stix2.v21.LanguageContent(
+        object_ref=CAMPAIGN_ID,
+        contents={
+            "en": {
+                "name": "the english text",
+            },
+        },
+    )

stix2/test/v21/test_location.py

@@ -5,6 +5,7 @@ import pytest
 import pytz
 
 import stix2
+import stix2.exceptions
 
 from .constants import LOCATION_ID
 
@@ -111,7 +112,7 @@ def test_parse_location(data):
     ],
 )
 def test_location_bad_latitude(data):
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(stix2.exceptions.InvalidValueError) as excinfo:
         stix2.parse(data)
 
     assert "Invalid value for Location 'latitude'" in str(excinfo.value)
@@ -140,7 +141,7 @@ def test_location_bad_latitude(data):
     ],
 )
 def test_location_bad_longitude(data):
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(stix2.exceptions.InvalidValueError) as excinfo:
         stix2.parse(data)
 
     assert "Invalid value for Location 'longitude'" in str(excinfo.value)
@@ -190,7 +191,7 @@ def test_location_properties_missing_when_precision_is_present(data):
     ],
 )
 def test_location_negative_precision(data):
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(stix2.exceptions.InvalidValueError) as excinfo:
         stix2.parse(data)
 
     assert "Invalid value for Location 'precision'" in str(excinfo.value)
@@ -264,6 +265,15 @@ def test_location_lat_or_lon_dependency_missing(data, msg):
     assert msg in str(excinfo.value)
 
 
+def test_location_complex_presence_constraint():
+    with pytest.raises(stix2.exceptions.PropertyPresenceError):
+        stix2.parse({
+            "type": "location",
+            "spec_version": "2.1",
+            "id": LOCATION_ID,
+        })
+
+
 def test_google_map_url_long_lat_provided():
     expected_url = "https://www.google.com/maps/search/?api=1&query=41.862401%2C-87.616001"
 

stix2/test/v21/test_malware.py

@@ -6,6 +6,7 @@ import pytz
 
 import stix2
 
+from ...exceptions import InvalidValueError, PropertyPresenceError
 from .constants import FAKE_TIME, MALWARE_ID, MALWARE_KWARGS
 
 EXPECTED_MALWARE = """{
@@ -17,7 +18,8 @@ EXPECTED_MALWARE = """{
     "name": "Cryptolocker",
     "malware_types": [
         "ransomware"
-    ]
+    ],
+    "is_family": false
 }"""
 
 
@@ -31,6 +33,7 @@ def test_malware_with_all_required_properties():
         modified=now,
         malware_types=["ransomware"],
         name="Cryptolocker",
+        is_family=False,
     )
 
     assert str(mal) == EXPECTED_MALWARE
@@ -77,7 +80,7 @@ def test_malware_required_properties():
         stix2.v21.Malware()
 
     assert excinfo.value.cls == stix2.v21.Malware
-    assert excinfo.value.properties == ["malware_types", "name"]
+    assert excinfo.value.properties == ["is_family", "malware_types"]
 
 
 def test_malware_required_property_name():
@@ -85,7 +88,7 @@ def test_malware_required_property_name():
         stix2.v21.Malware(malware_types=['ransomware'])
 
     assert excinfo.value.cls == stix2.v21.Malware
-    assert excinfo.value.properties == ["name"]
+    assert excinfo.value.properties == ["is_family"]
 
 
 def test_cannot_assign_to_malware_attributes(malware):
@@ -115,6 +118,7 @@ def test_invalid_kwarg_to_malware():
             "modified": "2016-05-12T08:17:27.000Z",
             "malware_types": ["ransomware"],
             "name": "Cryptolocker",
+            "is_family": False,
         },
     ],
 )
@@ -128,11 +132,12 @@ def test_parse_malware(data):
     assert mal.modified == dt.datetime(2016, 5, 12, 8, 17, 27, tzinfo=pytz.utc)
     assert mal.malware_types == ['ransomware']
     assert mal.name == 'Cryptolocker'
+    assert not mal.is_family
 
 
 def test_parse_malware_invalid_labels():
     data = re.compile('\\[.+\\]', re.DOTALL).sub('1', EXPECTED_MALWARE)
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(InvalidValueError) as excinfo:
         stix2.parse(data)
     assert "Invalid value for Malware 'malware_types'" in str(excinfo.value)
 
@@ -164,3 +169,31 @@ def test_parse_malware_clean_kill_chain_phases():
     data = EXPECTED_MALWARE.replace('2.1"', '2.1",%s' % kill_chain)
     mal = stix2.parse(data, version="2.1")
     assert mal['kill_chain_phases'][0]['phase_name'] == "1"
+
+
+def test_malware_invalid_last_before_first():
+    with pytest.raises(ValueError) as excinfo:
+        stix2.v21.Malware(first_seen="2017-01-01T12:34:56.000Z", last_seen="2017-01-01T12:33:56.000Z", **MALWARE_KWARGS)
+
+    assert "'last_seen' must be greater than or equal to 'first_seen'" in str(excinfo.value)
+
+
+def test_malware_family_no_name():
+    with pytest.raises(PropertyPresenceError):
+        stix2.parse({
+            "type": "malware",
+            "id": MALWARE_ID,
+            "spec_version": "2.1",
+            "is_family": True,
+            "malware_types": ["a type"],
+        })
+
+
+def test_malware_non_family_no_name():
+    stix2.parse({
+        "type": "malware",
+        "id": MALWARE_ID,
+        "spec_version": "2.1",
+        "is_family": False,
+        "malware_types": ["something"],
+    })

stix2/test/v21/test_malware_analysis.py

@@ -0,0 +1,84 @@
+import json
+
+import pytest
+
+import stix2.exceptions
+import stix2.utils
+import stix2.v21
+
+MALWARE_ANALYSIS_JSON = """{
+    "type": "malware-analysis",
+    "spec_version": "2.1",
+    "id": "malware-analysis--f8afc020-f92f-4906-a971-88ee5882eb46",
+    "created": "2017-11-28T09:44:58.418Z",
+    "modified": "2017-12-31T21:27:49.754Z",
+    "created_by_ref": "identity--e0353ed3-991e-4f71-a332-114c2f10b84f",
+    "labels": [
+        "label1",
+        "label2"
+    ],
+    "product": "Acme Malware Analyzer",
+    "version": "2.5",
+    "host_vm_ref": "software--1bda7336-fe67-469f-a8ca-ab6268b0449b",
+    "operating_system_ref": "software--c96bfaef-861b-408b-b0f1-b685881725ef",
+    "installed_software_refs": [
+        "software--7325bf2d-de9e-441e-b3b3-63df43149897",
+        "software--46a6a91d-1160-4867-a4d1-b14e080e4e5b"
+    ],
+    "configuration_version": "1.7",
+    "modules": [
+        "Super Analyzer"
+    ],
+    "analysis_engine_version": "1.2",
+    "analysis_definition_version": "3.4",
+    "submitted": "2018-11-23T06:45:55.747Z",
+    "analysis_started": "2018-11-29T07:30:03.895Z",
+    "analysis_ended": "2018-11-29T08:30:03.895Z",
+    "av_result": "malicious",
+    "analysis_sco_refs": [
+        "file--fc27e371-6c88-4c5c-868a-4dda0e60b167",
+        "url--6f7a74cd-8eb2-4b88-a4da-aa878e50ac2e"
+    ]
+}"""
+
+
+MALWARE_ANALYSIS_DICT = json.loads(MALWARE_ANALYSIS_JSON)
+
+
+def test_malware_analysis_example():
+    ma = stix2.v21.MalwareAnalysis(**MALWARE_ANALYSIS_DICT)
+
+    assert str(ma) == MALWARE_ANALYSIS_JSON
+
+
+@pytest.mark.parametrize(
+    "data", [
+        MALWARE_ANALYSIS_JSON,
+        MALWARE_ANALYSIS_DICT,
+    ],
+)
+def test_parse_malware_analysis(data):
+    ma = stix2.parse(data, version="2.1")
+
+    # timestamp-valued attributes whose values (from JSON) can't be compared
+    # directly, since stix2 internally converts them to datetime objects.
+    ts_attrs = {
+        "created",
+        "modified",
+        "submitted",
+        "analysis_started",
+        "analysis_ended",
+    }
+
+    for attr_name, attr_value in MALWARE_ANALYSIS_DICT.items():
+        cmp_value = stix2.utils.parse_into_datetime(attr_value) \
+            if attr_name in ts_attrs else attr_value
+
+        assert getattr(ma, attr_name) == cmp_value
+
+
+def test_malware_analysis_constraint():
+    with pytest.raises(stix2.exceptions.AtLeastOnePropertyError):
+        stix2.v21.MalwareAnalysis(
+            product="Acme Malware Analyzer",
+        )

stix2/test/v21/test_markings.py

@@ -25,7 +25,7 @@ EXPECTED_STATEMENT_MARKING_DEFINITION = """{
     "type": "marking-definition",
     "spec_version": "2.1",
     "id": "marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9",
-    "created": "2017-01-20T00:00:00Z",
+    "created": "2017-01-20T00:00:00.000Z",
     "definition_type": "statement",
     "definition": {
         "statement": "Copyright 2016, Example Corp"

stix2/test/v21/test_object_markings.py

@@ -3,6 +3,7 @@ import pytest
 from stix2 import exceptions, markings
 from stix2.v21 import TLP_AMBER, Malware
 
+from ...exceptions import MarkingNotFoundError
 from .constants import FAKE_TIME, MALWARE_ID
 from .constants import MALWARE_KWARGS as MALWARE_KWARGS_CONST
 from .constants import MARKING_IDS
@@ -349,7 +350,7 @@ def test_remove_markings_bad_markings():
         object_marking_refs=[MARKING_IDS[0], MARKING_IDS[1], MARKING_IDS[2]],
         **MALWARE_KWARGS
     )
-    with pytest.raises(AssertionError) as excinfo:
+    with pytest.raises(MarkingNotFoundError) as excinfo:
         markings.remove_markings(before, [MARKING_IDS[4]], None)
     assert str(excinfo.value) == "Marking ['%s'] was not found in Malware!" % MARKING_IDS[4]
 

stix2/test/v21/test_observed_data.py

@@ -1,10 +1,12 @@
 import datetime as dt
 import re
+import uuid
 
 import pytest
 import pytz
 
 import stix2
+import stix2.exceptions
 
 from .constants import IDENTITY_ID, OBSERVED_DATA_ID
 
@@ -24,6 +26,8 @@ EXPECTED = """{
     "objects": {
         "0": {
             "type": "file",
+            "id": "file--5956efbb-a7b0-566d-a7f9-a202eb05c70f",
+            "spec_version": "2.1",
             "name": "foo.exe"
         }
     }
@@ -41,13 +45,19 @@ def test_observed_data_example():
         number_observed=50,
         objects={
             "0": {
-                "name": "foo.exe",
                 "type": "file",
+                "id": "file--5956efbb-a7b0-566d-a7f9-a202eb05c70f",
+                "name": "foo.exe",
             },
         },
     )
 
-    assert str(observed_data) == EXPECTED
+    assert observed_data.id == "observed-data--b67d30ff-02ac-498a-92f9-32f845f448cf"
+    assert observed_data.created_by_ref == "identity--311b2d2d-f010-4473-83ec-1edf84858f4c"
+    assert observed_data.created == observed_data.modified == dt.datetime(2016, 4, 6, 19, 58, 16, tzinfo=pytz.utc)
+    assert observed_data.first_observed == observed_data.last_observed == dt.datetime(2015, 12, 21, 19, 00, 00, tzinfo=pytz.utc)
+    assert observed_data.number_observed == 50
+    assert observed_data.objects['0'] == stix2.v21.File(name="foo.exe")
 
 
 EXPECTED_WITH_REF = """{
@@ -63,13 +73,17 @@ EXPECTED_WITH_REF = """{
     "objects": {
         "0": {
             "type": "file",
+            "id": "file--5956efbb-a7b0-566d-a7f9-a202eb05c70f",
+            "spec_version": "2.1",
             "name": "foo.exe"
         },
         "1": {
             "type": "directory",
+            "id": "directory--536a61a4-0934-516b-9aad-fcbb75e0583a",
+            "spec_version": "2.1",
             "path": "/usr/home",
             "contains_refs": [
-                "0"
+                "file--5956efbb-a7b0-566d-a7f9-a202eb05c70f"
             ]
         }
     }
@@ -87,18 +101,86 @@ def test_observed_data_example_with_refs():
         number_observed=50,
         objects={
             "0": {
-                "name": "foo.exe",
                 "type": "file",
+                "id": "file--5956efbb-a7b0-566d-a7f9-a202eb05c70f",
+                "name": "foo.exe",
             },
             "1": {
                 "type": "directory",
+                "id": "directory--536a61a4-0934-516b-9aad-fcbb75e0583a",
                 "path": "/usr/home",
-                "contains_refs": ["0"],
+                "contains_refs": ["file--5956efbb-a7b0-566d-a7f9-a202eb05c70f"],
             },
         },
     )
+    assert observed_data.id == "observed-data--b67d30ff-02ac-498a-92f9-32f845f448cf"
+    assert observed_data.created_by_ref == "identity--311b2d2d-f010-4473-83ec-1edf84858f4c"
+    assert observed_data.created == observed_data.modified == dt.datetime(2016, 4, 6, 19, 58, 16, tzinfo=pytz.utc)
+    assert observed_data.first_observed == observed_data.last_observed == dt.datetime(2015, 12, 21, 19, 00, 00, tzinfo=pytz.utc)
+    assert observed_data.number_observed == 50
+    assert observed_data.objects['0'] == stix2.v21.File(name="foo.exe")
+    assert observed_data.objects['1'] == stix2.v21.Directory(path="/usr/home", contains_refs=["file--5956efbb-a7b0-566d-a7f9-a202eb05c70f"])
 
-    assert str(observed_data) == EXPECTED_WITH_REF
+
+EXPECTED_OBJECT_REFS = """{
+    "type": "observed-data",
+    "spec_version": "2.1",
+    "id": "observed-data--b67d30ff-02ac-498a-92f9-32f845f448cf",
+    "created_by_ref": "identity--f431f809-377b-45e0-aa1c-6a4751cae5ff",
+    "created": "2016-04-06T19:58:16.000Z",
+    "modified": "2016-04-06T19:58:16.000Z",
+    "first_observed": "2015-12-21T19:00:00Z",
+    "last_observed": "2015-12-21T19:00:00Z",
+    "number_observed": 50,
+    "object_refs": [
+        "file--758bf2c0-a6f1-56d1-872e-6b727467739a",
+        "url--d97ed5c4-3f33-46d9-b25b-c3d7b94d1457",
+        "mutex--eca0b3ba-8d76-11e9-a1fd-34415dabec0c"
+    ]
+}"""
+
+
+def test_observed_data_example_with_object_refs():
+    observed_data = stix2.v21.ObservedData(
+        id="observed-data--b67d30ff-02ac-498a-92f9-32f845f448cf",
+        created_by_ref="identity--f431f809-377b-45e0-aa1c-6a4751cae5ff",
+        created="2016-04-06T19:58:16.000Z",
+        modified="2016-04-06T19:58:16.000Z",
+        first_observed="2015-12-21T19:00:00Z",
+        last_observed="2015-12-21T19:00:00Z",
+        number_observed=50,
+        object_refs=[
+            "file--758bf2c0-a6f1-56d1-872e-6b727467739a",
+            "url--d97ed5c4-3f33-46d9-b25b-c3d7b94d1457",
+            "mutex--eca0b3ba-8d76-11e9-a1fd-34415dabec0c",
+        ],
+    )
+
+    assert str(observed_data) == EXPECTED_OBJECT_REFS
+
+
+def test_observed_data_object_constraint():
+    with pytest.raises(stix2.exceptions.MutuallyExclusivePropertiesError):
+        stix2.v21.ObservedData(
+            id="observed-data--b67d30ff-02ac-498a-92f9-32f845f448cf",
+            created_by_ref="identity--f431f809-377b-45e0-aa1c-6a4751cae5ff",
+            created="2016-04-06T19:58:16.000Z",
+            modified="2016-04-06T19:58:16.000Z",
+            first_observed="2015-12-21T19:00:00Z",
+            last_observed="2015-12-21T19:00:00Z",
+            number_observed=50,
+            objects={
+                "0": {
+                    "name": "foo.exe",
+                    "type": "file",
+                },
+            },
+            object_refs=[
+                "file--758bf2c0-a6f1-56d1-872e-6b727467739a",
+                "url--d97ed5c4-3f33-46d9-b25b-c3d7b94d1457",
+                "mutex--eca0b3ba-8d76-11e9-a1fd-34415dabec0c",
+            ],
+        )
 
 
 def test_observed_data_example_with_bad_refs():
@@ -114,19 +196,20 @@ def test_observed_data_example_with_bad_refs():
             objects={
                 "0": {
                     "type": "file",
+                    "id": "file--5956efbb-a7b0-566d-a7f9-a202eb05c70f",
                     "name": "foo.exe",
                 },
                 "1": {
                     "type": "directory",
                     "path": "/usr/home",
-                    "contains_refs": ["2"],
+                    "contains_refs": ["monkey--5956efbb-a7b0-566d-a7f9-a202eb05c70f"],
                 },
             },
         )
 
-    assert excinfo.value.cls == stix2.v21.ObservedData
-    assert excinfo.value.prop_name == "objects"
-    assert excinfo.value.reason == "Invalid object reference for 'Directory:contains_refs': '2' is not a valid object in local scope"
+    assert excinfo.value.cls == stix2.v21.Directory
+    assert excinfo.value.prop_name == "contains_refs"
+    assert "The type-specifying prefix 'monkey--' for this property is not valid" in excinfo.value.reason
 
 
 def test_observed_data_example_with_non_dictionary():
@@ -182,6 +265,7 @@ def test_observed_data_example_with_empty_dictionary():
                 "0": {
                     "name": "foo.exe",
                     "type": "file",
+                    "id": "file--5956efbb-a7b0-566d-a7f9-a202eb05c70f",
                 },
             },
         },
@@ -243,7 +327,7 @@ def test_parse_artifact_valid(data):
 )
 def test_parse_artifact_invalid(data):
     odata_str = OBJECTS_REGEX.sub('"objects": { %s }' % data, EXPECTED)
-    with pytest.raises(ValueError):
+    with pytest.raises(stix2.exceptions.InvalidValueError):
         stix2.parse(odata_str, version="2.1")
 
 
@@ -280,17 +364,21 @@ def test_parse_autonomous_system_valid(data):
         "type": "email-addr",
         "value": "john@example.com",
         "display_name": "John Doe",
-        "belongs_to_ref": "0"
+        "belongs_to_ref": "user-account--fc07c1af-6b11-41f8-97a4-47920d866a91"
     }""",
     ],
 )
 def test_parse_email_address(data):
-    odata = stix2.parse_observable(data, {"0": "user-account"}, version='2.1')
+    odata = stix2.parse_observable(data, version='2.1')
     assert odata.type == "email-addr"
 
-    odata_str = re.compile('"belongs_to_ref": "0"', re.DOTALL).sub('"belongs_to_ref": "3"', data)
-    with pytest.raises(stix2.exceptions.InvalidObjRefError):
-        stix2.parse_observable(odata_str, {"0": "user-account"}, version='2.1')
+    odata_str = re.compile(
+        '"belongs_to_ref": "user-account--fc07c1af-6b11-41f8-97a4-47920d866a91"', re.DOTALL,
+    ).sub(
+        '"belongs_to_ref": "mutex--9be6365f-b89c-48c0-9340-6953f6595718"', data,
+    )
+    with pytest.raises(stix2.exceptions.InvalidValueError):
+        stix2.parse_observable(odata_str, version='2.1')
 
 
 @pytest.mark.parametrize(
@@ -301,12 +389,12 @@ def test_parse_email_address(data):
         "is_multipart": true,
         "content_type": "multipart/mixed",
         "date": "2016-06-19T14:20:40.000Z",
-        "from_ref": "1",
+        "from_ref": "email-addr--d4ef7e1f-086d-5ff4-bce4-312ddc3eae76",
         "to_refs": [
-          "2"
+          "email-addr--8b0eb924-208c-5efd-80e5-84e2d610e54b"
         ],
         "cc_refs": [
-          "3"
+          "email-addr--1766f860-5cf3-5697-8789-35f1242663d5"
         ],
         "subject": "Check out this picture of a cat!",
         "additional_header_fields": {
@@ -323,12 +411,12 @@ def test_parse_email_address(data):
           {
             "content_type": "image/png",
             "content_disposition": "attachment; filename=\\"tabby.png\\"",
-            "body_raw_ref": "4"
+            "body_raw_ref": "artifact--80b04ad8-db52-464b-a85a-a44a5f3a60c5"
           },
           {
             "content_type": "application/zip",
             "content_disposition": "attachment; filename=\\"tabby_pics.zip\\"",
-            "body_raw_ref": "5"
+            "body_raw_ref": "file--e63474fc-b386-5630-a003-1b555e22f99b"
           }
         ]
     }
@@ -336,15 +424,7 @@ def test_parse_email_address(data):
     ],
 )
 def test_parse_email_message(data):
-    valid_refs = {
-        "0": "email-message",
-        "1": "email-addr",
-        "2": "email-addr",
-        "3": "email-addr",
-        "4": "artifact",
-        "5": "file",
-    }
-    odata = stix2.parse_observable(data, valid_refs, version='2.1')
+    odata = stix2.parse_observable(data, version='2.1')
     assert odata.type == "email-message"
     assert odata.body_multipart[0].content_disposition == "inline"
 
@@ -354,8 +434,8 @@ def test_parse_email_message(data):
         """
     {
          "type": "email-message",
-         "from_ref": "0",
-         "to_refs": ["1"],
+         "from_ref": "email-addr--d4ef7e1f-086d-5ff4-bce4-312ddc3eae76",
+         "to_refs": ["email-addr--8b0eb924-208c-5efd-80e5-84e2d610e54b"],
          "is_multipart": true,
          "date": "1997-11-21T15:55:06.000Z",
          "subject": "Saying Hello",
@@ -365,12 +445,8 @@ def test_parse_email_message(data):
     ],
 )
 def test_parse_email_message_not_multipart(data):
-    valid_refs = {
-        "0": "email-addr",
-        "1": "email-addr",
-    }
     with pytest.raises(stix2.exceptions.DependentPropertiesError) as excinfo:
-        stix2.parse_observable(data, valid_refs, version='2.1')
+        stix2.parse_observable(data, version='2.1')
 
     assert excinfo.value.cls == stix2.v21.EmailMessage
     assert excinfo.value.dependencies == [("is_multipart", "body")]
@@ -380,35 +456,38 @@ def test_parse_email_message_not_multipart(data):
     "data", [
         """"0": {
             "type": "file",
+            "id": "file--ecd47d73-15e4-5250-afda-ef8897b22340",
             "hashes": {
                 "SHA-256": "ceafbfd424be2ca4a5f0402cae090dda2fb0526cf521b60b60077c0f622b285a"
             }
         },
         "1": {
             "type": "file",
+            "id": "file--65f2873d-38c2-56b4-bfa5-e3ef21e8a3c3",
             "hashes": {
-                "SHA-256": "19c549ec2628b989382f6b280cbd7bb836a0b461332c0fe53511ce7d584b89d3"
+                "SHA-1": "6e71b3cac15d32fe2d36c270887df9479c25c640"
             }
         },
         "2": {
             "type": "file",
+            "id": "file--ef2d6dca-ec7d-5ab7-8dd9-ec9c0dee0eac",
             "hashes": {
-                "SHA-256": "0969de02ecf8a5f003e3f6d063d848c8a193aada092623f8ce408c15bcb5f038"
+                "SHA-512": "b7e98c78c24fb4c2c7b175e90474b21eae0ccf1b5ea4708b4e0f2d2940004419edc7161c18a1e71b2565df099ba017bcaa67a248e2989b6268ce078b88f2e210"
             }
         },
         "3": {
             "type": "file",
             "name": "foo.zip",
             "hashes": {
-                "SHA-256": "35a01331e9ad96f751278b891b6ea09699806faedfa237d40513d92ad1b7100f"
+                "SHA3-256": "35a01331e9ad96f751278b891b6ea09699806faedfa237d40513d92ad1b7100f"
             },
             "mime_type": "application/zip",
             "extensions": {
                 "archive-ext": {
                     "contains_refs": [
-                        "0",
-                        "1",
-                        "2"
+                        "file--ecd47d73-15e4-5250-afda-ef8897b22340",
+                        "file--65f2873d-38c2-56b4-bfa5-e3ef21e8a3c3",
+                        "file--ef2d6dca-ec7d-5ab7-8dd9-ec9c0dee0eac"
                     ]
                 }
             }
@@ -419,7 +498,11 @@ def test_parse_file_archive(data):
     odata_str = OBJECTS_REGEX.sub('"objects": { %s }' % data, EXPECTED)
     odata = stix2.parse(odata_str, version="2.1")
     assert all(x in odata.objects["3"].extensions['archive-ext'].contains_refs
-               for x in ["0", "1", "2"])
+               for x in [
+                   "file--ecd47d73-15e4-5250-afda-ef8897b22340",
+                   "file--65f2873d-38c2-56b4-bfa5-e3ef21e8a3c3",
+                   "file--ef2d6dca-ec7d-5ab7-8dd9-ec9c0dee0eac",
+               ])
 
 
 @pytest.mark.parametrize(
@@ -430,12 +513,12 @@ def test_parse_file_archive(data):
         "is_multipart": true,
         "content_type": "multipart/mixed",
         "date": "2016-06-19T14:20:40.000Z",
-        "from_ref": "1",
+        "from_ref": "email-addr--d4ef7e1f-086d-5ff4-bce4-312ddc3eae76",
         "to_refs": [
-          "2"
+          "email-addr--8b0eb924-208c-5efd-80e5-84e2d610e54b"
         ],
         "cc_refs": [
-          "3"
+          "email-addr--1766f860-5cf3-5697-8789-35f1242663d5"
         ],
         "subject": "Check out this picture of a cat!",
         "additional_header_fields": {
@@ -456,7 +539,7 @@ def test_parse_file_archive(data):
           {
             "content_type": "application/zip",
             "content_disposition": "attachment; filename=\\"tabby_pics.zip\\"",
-            "body_raw_ref": "5"
+            "body_raw_ref": "file--e63474fc-b386-5630-a003-1b555e22f99b"
           }
         ]
     }
@@ -464,19 +547,10 @@ def test_parse_file_archive(data):
     ],
 )
 def test_parse_email_message_with_at_least_one_error(data):
-    valid_refs = {
-        "0": "email-message",
-        "1": "email-addr",
-        "2": "email-addr",
-        "3": "email-addr",
-        "4": "artifact",
-        "5": "file",
-    }
-    with pytest.raises(stix2.exceptions.AtLeastOnePropertyError) as excinfo:
-        stix2.parse_observable(data, valid_refs, version='2.1')
+    with pytest.raises(stix2.exceptions.InvalidValueError) as excinfo:
+        stix2.parse_observable(data, version='2.1')
 
-    assert excinfo.value.cls == stix2.v21.EmailMIMEComponent
-    assert excinfo.value.properties == ["body", "body_raw_ref"]
+    assert excinfo.value.cls == stix2.v21.EmailMessage
     assert "At least one of the" in str(excinfo.value)
     assert "must be populated" in str(excinfo.value)
 
@@ -486,8 +560,8 @@ def test_parse_email_message_with_at_least_one_error(data):
         """
     {
         "type": "network-traffic",
-        "src_ref": "0",
-        "dst_ref": "1",
+        "src_ref": "ipv4-addr--e535b017-cc1c-566b-a3e2-f69f92ed9c4c",
+        "dst_ref": "ipv4-addr--78327430-9ad9-5632-ae3d-8e2fce8f5483",
         "protocols": [
           "tcp"
         ]
@@ -497,13 +571,12 @@ def test_parse_email_message_with_at_least_one_error(data):
 )
 def test_parse_basic_tcp_traffic(data):
     odata = stix2.parse_observable(
-        data, {"0": "ipv4-addr", "1": "ipv4-addr"},
-        version='2.1',
+        data, version='2.1',
     )
 
     assert odata.type == "network-traffic"
-    assert odata.src_ref == "0"
-    assert odata.dst_ref == "1"
+    assert odata.src_ref == "ipv4-addr--e535b017-cc1c-566b-a3e2-f69f92ed9c4c"
+    assert odata.dst_ref == "ipv4-addr--78327430-9ad9-5632-ae3d-8e2fce8f5483"
     assert odata.protocols == ["tcp"]
 
 
@@ -521,7 +594,7 @@ def test_parse_basic_tcp_traffic(data):
         "src_byte_count": 35779,
         "dst_byte_count": 935750,
         "encapsulates_refs": [
-          "4"
+          "network-traffic--016914c3-b680-5df2-81c4-bb9ccf8dc8b0"
         ]
   }
     """,
@@ -529,7 +602,7 @@ def test_parse_basic_tcp_traffic(data):
 )
 def test_parse_basic_tcp_traffic_with_error(data):
     with pytest.raises(stix2.exceptions.AtLeastOnePropertyError) as excinfo:
-        stix2.parse_observable(data, {"4": "network-traffic"}, version='2.1')
+        stix2.parse_observable(data, version='2.1')
 
     assert excinfo.value.cls == stix2.v21.NetworkTraffic
     assert excinfo.value.properties == ["dst_ref", "src_ref"]
@@ -575,16 +648,18 @@ def test_observed_data_with_process_example():
         objects={
             "0": {
                 "type": "file",
+                "id": "file--0d16c8d3-c177-5f5d-a022-b1bdac329bea",
                 "hashes": {
                     "SHA-256": "35a01331e9ad96f751278b891b6ea09699806faedfa237d40513d92ad1b7100f",
                 },
             },
             "1": {
                 "type": "process",
+                "id": "process--f6c4a02c-23e1-4a6d-a0d7-d862e893817a",
                 "pid": 1221,
-                "created": "2016-01-20T14:11:25.55Z",
+                "created_time": "2016-01-20T14:11:25.55Z",
                 "command_line": "./gedit-bin --new-window",
-                "image_ref": "0",
+                "image_ref": "file--0d16c8d3-c177-5f5d-a022-b1bdac329bea",
             },
         },
     )
@@ -628,31 +703,33 @@ def test_artifact_mutual_exclusion_error():
 
 
 def test_directory_example():
-    dir = stix2.v21.Directory(
-        _valid_refs={"1": "file"},
-        path='/usr/lib',
-        created="2015-12-21T19:00:00Z",
-        modified="2015-12-24T19:00:00Z",
-        accessed="2015-12-21T20:00:00Z",
-        contains_refs=["1"],
+    f = stix2.v21.File(
+        name="penguin.exe",
     )
 
-    assert dir.path == '/usr/lib'
-    assert dir.created == dt.datetime(2015, 12, 21, 19, 0, 0, tzinfo=pytz.utc)
-    assert dir.modified == dt.datetime(2015, 12, 24, 19, 0, 0, tzinfo=pytz.utc)
-    assert dir.accessed == dt.datetime(2015, 12, 21, 20, 0, 0, tzinfo=pytz.utc)
-    assert dir.contains_refs == ["1"]
+    dir1 = stix2.v21.Directory(
+        path='/usr/lib',
+        ctime="2015-12-21T19:00:00Z",
+        mtime="2015-12-24T19:00:00Z",
+        atime="2015-12-21T20:00:00Z",
+        contains_refs=[str(f.id)],
+    )
+
+    assert dir1.path == '/usr/lib'
+    assert dir1.ctime == dt.datetime(2015, 12, 21, 19, 0, 0, tzinfo=pytz.utc)
+    assert dir1.mtime == dt.datetime(2015, 12, 24, 19, 0, 0, tzinfo=pytz.utc)
+    assert dir1.atime == dt.datetime(2015, 12, 21, 20, 0, 0, tzinfo=pytz.utc)
+    assert dir1.contains_refs == ["file--9d050a3b-72cd-5b57-bf18-024e74e1e5eb"]
 
 
 def test_directory_example_ref_error():
-    with pytest.raises(stix2.exceptions.InvalidObjRefError) as excinfo:
+    with pytest.raises(stix2.exceptions.InvalidValueError) as excinfo:
         stix2.v21.Directory(
-            _valid_refs=[],
             path='/usr/lib',
-            created="2015-12-21T19:00:00Z",
-            modified="2015-12-24T19:00:00Z",
-            accessed="2015-12-21T20:00:00Z",
-            contains_refs=["1"],
+            ctime="2015-12-21T19:00:00Z",
+            mtime="2015-12-24T19:00:00Z",
+            atime="2015-12-21T20:00:00Z",
+            contains_refs=["domain-name--02af94ea-7e38-5718-87c3-5cc023e3d49d"],
         )
 
     assert excinfo.value.cls == stix2.v21.Directory
@@ -660,22 +737,24 @@ def test_directory_example_ref_error():
 
 
 def test_domain_name_example():
-    dn = stix2.v21.DomainName(
-        _valid_refs={"1": 'domain-name'},
-        value="example.com",
-        resolves_to_refs=["1"],
+    dn1 = stix2.v21.DomainName(
+        value="mitre.org",
     )
 
-    assert dn.value == "example.com"
-    assert dn.resolves_to_refs == ["1"]
+    dn2 = stix2.v21.DomainName(
+        value="example.com",
+        resolves_to_refs=[str(dn1.id)],
+    )
+
+    assert dn2.value == "example.com"
+    assert dn2.resolves_to_refs == ["domain-name--02af94ea-7e38-5718-87c3-5cc023e3d49d"]
 
 
 def test_domain_name_example_invalid_ref_type():
-    with pytest.raises(stix2.exceptions.InvalidObjRefError) as excinfo:
+    with pytest.raises(stix2.exceptions.InvalidValueError) as excinfo:
         stix2.v21.DomainName(
-            _valid_refs={"1": "file"},
             value="example.com",
-            resolves_to_refs=["1"],
+            resolves_to_refs=["file--44a431e6-764b-5556-a3f5-bf655930a581"],
         )
 
     assert excinfo.value.cls == stix2.v21.DomainName
@@ -691,9 +770,9 @@ def test_file_example():
         size=100,
         magic_number_hex="1C",
         mime_type="application/msword",
-        created="2016-12-21T19:00:00Z",
-        modified="2016-12-24T19:00:00Z",
-        accessed="2016-12-21T20:00:00Z",
+        ctime="2016-12-21T19:00:00Z",
+        mtime="2016-12-24T19:00:00Z",
+        atime="2016-12-21T20:00:00Z",
     )
 
     assert f.name == "qwerty.dll"
@@ -701,9 +780,9 @@ def test_file_example():
     assert f.magic_number_hex == "1C"
     assert f.hashes["SHA-256"] == "ceafbfd424be2ca4a5f0402cae090dda2fb0526cf521b60b60077c0f622b285a"
     assert f.mime_type == "application/msword"
-    assert f.created == dt.datetime(2016, 12, 21, 19, 0, 0, tzinfo=pytz.utc)
-    assert f.modified == dt.datetime(2016, 12, 24, 19, 0, 0, tzinfo=pytz.utc)
-    assert f.accessed == dt.datetime(2016, 12, 21, 20, 0, 0, tzinfo=pytz.utc)
+    assert f.ctime == dt.datetime(2016, 12, 21, 19, 0, 0, tzinfo=pytz.utc)
+    assert f.mtime == dt.datetime(2016, 12, 24, 19, 0, 0, tzinfo=pytz.utc)
+    assert f.atime == dt.datetime(2016, 12, 21, 20, 0, 0, tzinfo=pytz.utc)
 
 
 def test_file_example_with_NTFSExt():
@@ -726,7 +805,7 @@ def test_file_example_with_NTFSExt():
 
 
 def test_file_example_with_empty_NTFSExt():
-    with pytest.raises(stix2.exceptions.AtLeastOnePropertyError) as excinfo:
+    with pytest.raises(stix2.exceptions.InvalidValueError) as excinfo:
         stix2.v21.File(
             name="abc.txt",
             extensions={
@@ -734,8 +813,7 @@ def test_file_example_with_empty_NTFSExt():
             },
         )
 
-    assert excinfo.value.cls == stix2.v21.NTFSExt
-    assert excinfo.value.properties == sorted(list(stix2.NTFSExt._properties.keys()))
+    assert excinfo.value.cls == stix2.v21.File
 
 
 def test_file_example_with_PDFExt():
@@ -818,6 +896,7 @@ RASTER_IMAGE_EXT = """{
 "objects": {
   "0": {
     "type": "file",
+    "id": "file--44a431e6-764b-5556-a3f5-bf655930a581",
     "name": "picture.jpg",
     "hashes": {
       "SHA-256": "35a01331e9ad96f751278b891b6ea09699806faedfa237d40513d92ad1b7100f"
@@ -929,18 +1008,17 @@ def test_file_example_encryption_error():
     assert "At least one of the (hashes, name)" in str(excinfo.value)
 
 
-def test_ip4_address_example():
+def test_ipv4_address_example():
     ip4 = stix2.v21.IPv4Address(
-        _valid_refs={"4": "mac-addr", "5": "mac-addr"},
         value="198.51.100.3",
-        resolves_to_refs=["4", "5"],
+        resolves_to_refs=["mac-addr--a85820f7-d9b7-567a-a3a6-dedc34139342", "mac-addr--9a59b496-fdeb-510f-97b5-7137210bc699"],
     )
 
     assert ip4.value == "198.51.100.3"
-    assert ip4.resolves_to_refs == ["4", "5"]
+    assert ip4.resolves_to_refs == ["mac-addr--a85820f7-d9b7-567a-a3a6-dedc34139342", "mac-addr--9a59b496-fdeb-510f-97b5-7137210bc699"]
 
 
-def test_ip4_address_valid_refs():
+def test_ipv4_address_valid_refs():
     mac1 = stix2.v21.MACAddress(
         value="a1:b2:c3:d4:e5:f6",
     )
@@ -949,22 +1027,21 @@ def test_ip4_address_valid_refs():
     )
 
     ip4 = stix2.v21.IPv4Address(
-        _valid_refs={"1": mac1, "2": mac2},
         value="177.60.40.7",
-        resolves_to_refs=["1", "2"],
+        resolves_to_refs=[str(mac1.id), str(mac2.id)],
     )
 
     assert ip4.value == "177.60.40.7"
-    assert ip4.resolves_to_refs == ["1", "2"]
+    assert ip4.resolves_to_refs == ["mac-addr--a85820f7-d9b7-567a-a3a6-dedc34139342", "mac-addr--9a59b496-fdeb-510f-97b5-7137210bc699"]
 
 
-def test_ip4_address_example_cidr():
+def test_ipv4_address_example_cidr():
     ip4 = stix2.v21.IPv4Address(value="198.51.100.0/24")
 
     assert ip4.value == "198.51.100.0/24"
 
 
-def test_ip6_address_example():
+def test_ipv6_address_example():
     ip6 = stix2.v21.IPv6Address(value="2001:0db8:85a3:0000:0000:8a2e:0370:7334")
 
     assert ip6.value == "2001:0db8:85a3:0000:0000:8a2e:0370:7334"
@@ -978,14 +1055,13 @@ def test_mac_address_example():
 
 def test_network_traffic_example():
     nt = stix2.v21.NetworkTraffic(
-        _valid_refs={"0": "ipv4-addr", "1": "ipv4-addr"},
-        protocols="tcp",
-        src_ref="0",
-        dst_ref="1",
+        protocols=["tcp"],
+        src_ref="ipv4-addr--29a591d9-533a-5ecd-a5a1-cadee4411e88",
+        dst_ref="ipv4-addr--6d39dd0b-1f74-5faf-8d76-d8762c2a57cb",
     )
     assert nt.protocols == ["tcp"]
-    assert nt.src_ref == "0"
-    assert nt.dst_ref == "1"
+    assert nt.src_ref == "ipv4-addr--29a591d9-533a-5ecd-a5a1-cadee4411e88"
+    assert nt.dst_ref == "ipv4-addr--6d39dd0b-1f74-5faf-8d76-d8762c2a57cb"
 
 
 def test_network_traffic_http_request_example():
@@ -1000,9 +1076,8 @@ def test_network_traffic_http_request_example():
         },
     )
     nt = stix2.v21.NetworkTraffic(
-        _valid_refs={"0": "ipv4-addr"},
-        protocols="tcp",
-        src_ref="0",
+        protocols=["tcp"],
+        src_ref="ipv4-addr--29a591d9-533a-5ecd-a5a1-cadee4411e88",
         extensions={'http-request-ext': h},
     )
     assert nt.extensions['http-request-ext'].request_method == "get"
@@ -1016,9 +1091,8 @@ def test_network_traffic_http_request_example():
 def test_network_traffic_icmp_example():
     h = stix2.v21.ICMPExt(icmp_type_hex="08", icmp_code_hex="00")
     nt = stix2.v21.NetworkTraffic(
-        _valid_refs={"0": "ipv4-addr"},
-        protocols="tcp",
-        src_ref="0",
+        protocols=["tcp"],
+        src_ref="ipv4-addr--29a591d9-533a-5ecd-a5a1-cadee4411e88",
         extensions={'icmp-ext': h},
     )
     assert nt.extensions['icmp-ext'].icmp_type_hex == "08"
@@ -1033,9 +1107,8 @@ def test_network_traffic_socket_example():
         socket_type="SOCK_STREAM",
     )
     nt = stix2.v21.NetworkTraffic(
-        _valid_refs={"0": "ipv4-addr"},
-        protocols="tcp",
-        src_ref="0",
+        protocols=["tcp"],
+        src_ref="ipv4-addr--29a591d9-533a-5ecd-a5a1-cadee4411e88",
         extensions={'socket-ext': h},
     )
     assert nt.extensions['socket-ext'].is_listening
@@ -1047,9 +1120,8 @@ def test_network_traffic_socket_example():
 def test_network_traffic_tcp_example():
     h = stix2.v21.TCPExt(src_flags_hex="00000002")
     nt = stix2.v21.NetworkTraffic(
-        _valid_refs={"0": "ipv4-addr"},
-        protocols="tcp",
-        src_ref="0",
+        protocols=["tcp"],
+        src_ref="ipv4-addr--29a591d9-533a-5ecd-a5a1-cadee4411e88",
         extensions={'tcp-ext': h},
     )
     assert nt.extensions['tcp-ext'].src_flags_hex == "00000002"
@@ -1063,11 +1135,10 @@ def test_mutex_example():
 
 def test_process_example():
     p = stix2.v21.Process(
-        _valid_refs={"0": "file"},
         pid=1221,
-        created="2016-01-20T14:11:25.55Z",
+        created_time="2016-01-20T14:11:25.55Z",
         command_line="./gedit-bin --new-window",
-        image_ref="0",
+        image_ref="file--ea587d87-5ed2-5625-a9ac-01fd64161fd8",
     )
 
     assert p.command_line == "./gedit-bin --new-window"
@@ -1079,7 +1150,7 @@ def test_process_example_empty_error():
 
     assert excinfo.value.cls == stix2.v21.Process
     properties_of_process = list(stix2.v21.Process._properties.keys())
-    properties_of_process.remove("type")
+    properties_of_process = [prop for prop in properties_of_process if prop not in ["type", "id", "defanged", "spec_version"]]
     assert excinfo.value.properties == sorted(properties_of_process)
     msg = "At least one of the ({1}) properties for {0} must be populated."
     msg = msg.format(
@@ -1090,14 +1161,12 @@ def test_process_example_empty_error():
 
 
 def test_process_example_empty_with_extensions():
-    with pytest.raises(stix2.exceptions.AtLeastOnePropertyError) as excinfo:
+    with pytest.raises(stix2.exceptions.InvalidValueError) as excinfo:
         stix2.v21.Process(extensions={
             "windows-process-ext": {},
         })
 
-    assert excinfo.value.cls == stix2.v21.WindowsProcessExt
-    properties_of_extension = list(stix2.v21.WindowsProcessExt._properties.keys())
-    assert excinfo.value.properties == sorted(properties_of_extension)
+    assert excinfo.value.cls == stix2.v21.Process
 
 
 def test_process_example_windows_process_ext():
@@ -1119,7 +1188,7 @@ def test_process_example_windows_process_ext():
 
 
 def test_process_example_windows_process_ext_empty():
-    with pytest.raises(stix2.exceptions.AtLeastOnePropertyError) as excinfo:
+    with pytest.raises(stix2.exceptions.InvalidValueError) as excinfo:
         stix2.v21.Process(
             pid=1221,
             extensions={
@@ -1127,9 +1196,7 @@ def test_process_example_windows_process_ext_empty():
             },
         )
 
-    assert excinfo.value.cls == stix2.v21.WindowsProcessExt
-    properties_of_extension = list(stix2.v21.WindowsProcessExt._properties.keys())
-    assert excinfo.value.properties == sorted(properties_of_extension)
+    assert excinfo.value.cls == stix2.v21.Process
 
 
 def test_process_example_extensions_empty():
@@ -1262,7 +1329,7 @@ def test_user_account_unix_account_ext_example():
 
 
 def test_windows_registry_key_example():
-    with pytest.raises(ValueError):
+    with pytest.raises(stix2.exceptions.InvalidValueError):
         stix2.v21.WindowsRegistryValueType(
             name="Foo",
             data="qwerty",
@@ -1307,15 +1374,155 @@ def test_new_version_with_related_objects():
         objects={
             'src_ip': {
                 'type': 'ipv4-addr',
+                'id': 'ipv4-addr--2b94bc65-17d4-54f6-9ffe-7d103551bb9f',
                 'value': '127.0.0.1/32',
             },
             'domain': {
                 'type': 'domain-name',
+                'id': 'domain-name--220a2699-5ebf-5b57-bf02-424964bb19c0',
                 'value': 'example.com',
-                'resolves_to_refs': ['src_ip'],
+                'resolves_to_refs': ['ipv4-addr--2b94bc65-17d4-54f6-9ffe-7d103551bb9f'],
             },
         },
     )
     new_version = data.new_version(last_observed="2017-12-12T12:00:00Z")
     assert new_version.last_observed.year == 2017
-    assert new_version.objects['domain'].resolves_to_refs[0] == 'src_ip'
+    assert new_version.objects['domain'].resolves_to_refs[0] == 'ipv4-addr--2b94bc65-17d4-54f6-9ffe-7d103551bb9f'
+
+
+def test_objects_deprecation():
+    with pytest.warns(stix2.exceptions.STIXDeprecationWarning):
+
+        stix2.v21.ObservedData(
+            first_observed="2016-03-12T12:00:00Z",
+            last_observed="2016-03-12T12:00:00Z",
+            number_observed=1,
+            objects={
+                "0": {
+                    "type": "file",
+                    "name": "foo",
+                },
+            },
+        )
+
+
+def test_deterministic_id_same_extra_prop_vals():
+    email_addr_1 = stix2.v21.EmailAddress(
+        value="john@example.com",
+        display_name="Johnny Doe",
+    )
+
+    email_addr_2 = stix2.v21.EmailAddress(
+        value="john@example.com",
+        display_name="Johnny Doe",
+    )
+
+    assert email_addr_1.id == email_addr_2.id
+
+    uuid_obj_1 = uuid.UUID(email_addr_1.id[-36:])
+    assert uuid_obj_1.variant == uuid.RFC_4122
+    assert uuid_obj_1.version == 5
+
+    uuid_obj_2 = uuid.UUID(email_addr_2.id[-36:])
+    assert uuid_obj_2.variant == uuid.RFC_4122
+    assert uuid_obj_2.version == 5
+
+
+def test_deterministic_id_diff_extra_prop_vals():
+    email_addr_1 = stix2.v21.EmailAddress(
+        value="john@example.com",
+        display_name="Johnny Doe",
+    )
+
+    email_addr_2 = stix2.v21.EmailAddress(
+        value="john@example.com",
+        display_name="Janey Doe",
+    )
+
+    assert email_addr_1.id == email_addr_2.id
+
+    uuid_obj_1 = uuid.UUID(email_addr_1.id[-36:])
+    assert uuid_obj_1.variant == uuid.RFC_4122
+    assert uuid_obj_1.version == 5
+
+    uuid_obj_2 = uuid.UUID(email_addr_2.id[-36:])
+    assert uuid_obj_2.variant == uuid.RFC_4122
+    assert uuid_obj_2.version == 5
+
+
+def test_deterministic_id_diff_contributing_prop_vals():
+    email_addr_1 = stix2.v21.EmailAddress(
+        value="john@example.com",
+        display_name="Johnny Doe",
+    )
+
+    email_addr_2 = stix2.v21.EmailAddress(
+        value="jane@example.com",
+        display_name="Janey Doe",
+    )
+
+    assert email_addr_1.id != email_addr_2.id
+
+    uuid_obj_1 = uuid.UUID(email_addr_1.id[-36:])
+    assert uuid_obj_1.variant == uuid.RFC_4122
+    assert uuid_obj_1.version == 5
+
+    uuid_obj_2 = uuid.UUID(email_addr_2.id[-36:])
+    assert uuid_obj_2.variant == uuid.RFC_4122
+    assert uuid_obj_2.version == 5
+
+
+def test_deterministic_id_no_contributing_props():
+    email_msg_1 = stix2.v21.EmailMessage(
+        is_multipart=False,
+    )
+
+    email_msg_2 = stix2.v21.EmailMessage(
+        is_multipart=False,
+    )
+
+    assert email_msg_1.id != email_msg_2.id
+
+    uuid_obj_1 = uuid.UUID(email_msg_1.id[-36:])
+    assert uuid_obj_1.variant == uuid.RFC_4122
+    assert uuid_obj_1.version == 4
+
+    uuid_obj_2 = uuid.UUID(email_msg_2.id[-36:])
+    assert uuid_obj_2.variant == uuid.RFC_4122
+    assert uuid_obj_2.version == 4
+
+
+def test_ipv4_resolves_to_refs_deprecation():
+    with pytest.warns(stix2.exceptions.STIXDeprecationWarning):
+
+        stix2.v21.IPv4Address(
+            value="26.09.19.70",
+            resolves_to_refs=["mac-addr--08900593-0265-52fc-93c0-5b4a942f5887"],
+        )
+
+
+def test_ipv4_belongs_to_refs_deprecation():
+    with pytest.warns(stix2.exceptions.STIXDeprecationWarning):
+
+        stix2.v21.IPv4Address(
+            value="21.12.19.64",
+            belongs_to_refs=["autonomous-system--52e0a49d-d683-5801-a7b8-145765a1e116"],
+        )
+
+
+def test_ipv6_resolves_to_refs_deprecation():
+    with pytest.warns(stix2.exceptions.STIXDeprecationWarning):
+
+        stix2.v21.IPv6Address(
+            value="2001:0db8:85a3:0000:0000:8a2e:0370:7334",
+            resolves_to_refs=["mac-addr--08900593-0265-52fc-93c0-5b4a942f5887"],
+        )
+
+
+def test_ipv6_belongs_to_refs_deprecation():
+    with pytest.warns(stix2.exceptions.STIXDeprecationWarning):
+
+        stix2.v21.IPv6Address(
+            value="2001:0db8:85a3:0000:0000:8a2e:0370:7334",
+            belongs_to_refs=["autonomous-system--52e0a49d-d683-5801-a7b8-145765a1e116"],
+        )

stix2/test/v21/test_opinion.py

@@ -23,10 +23,10 @@ EXPECTED_OPINION = """{
     "created": "2016-05-12T08:17:27.000Z",
     "modified": "2016-05-12T08:17:27.000Z",
     "explanation": "%s",
+    "opinion": "strongly-disagree",
     "object_refs": [
         "relationship--16d2358f-3b0d-4c88-b047-0da2f7ed4471"
-    ],
-    "opinion": "strongly-disagree"
+    ]
 }""" % EXPLANATION
 
 EXPECTED_OPINION_REPR = "Opinion(" + " ".join((
@@ -37,8 +37,9 @@ EXPECTED_OPINION_REPR = "Opinion(" + " ".join((
     created='2016-05-12T08:17:27.000Z',
     modified='2016-05-12T08:17:27.000Z',
     explanation="%s",
-    object_refs=['relationship--16d2358f-3b0d-4c88-b047-0da2f7ed4471'],
-    opinion='strongly-disagree'""" % EXPLANATION
+    opinion='strongly-disagree',
+    object_refs=['relationship--16d2358f-3b0d-4c88-b047-0da2f7ed4471']
+    """ % EXPLANATION
 ).split()) + ")"
 
 

stix2/test/v21/test_properties.py

@@ -1,11 +1,11 @@
-import uuid
-
 import pytest
 
 import stix2
-from stix2.exceptions import AtLeastOnePropertyError, DictionaryKeyError
+from stix2.exceptions import (
+    AtLeastOnePropertyError, CustomContentError, DictionaryKeyError,
+)
 from stix2.properties import (
-    ERROR_INVALID_ID, BinaryProperty, BooleanProperty, DictionaryProperty,
+    BinaryProperty, BooleanProperty, DictionaryProperty,
     EmbeddedObjectProperty, EnumProperty, ExtensionsProperty, FloatProperty,
     HashesProperty, HexProperty, IDProperty, IntegerProperty, ListProperty,
     Property, ReferenceProperty, StringProperty, TimestampProperty,
@@ -89,7 +89,7 @@ def test_type_property():
     assert prop.clean(prop.default())
 
 
-ID_PROP = IDProperty('my-type')
+ID_PROP = IDProperty('my-type', spec_version="2.1")
 MY_ID = 'my-type--232c9d3f-49fc-4440-bb01-607f638778e7'
 
 
@@ -127,7 +127,7 @@ CONSTANT_IDS.extend(constants.RELATIONSHIP_IDS)
 @pytest.mark.parametrize("value", CONSTANT_IDS)
 def test_id_property_valid_for_type(value):
     type = value.split('--', 1)[0]
-    assert IDProperty(type=type).clean(value) == value
+    assert IDProperty(type=type, spec_version="2.1").clean(value) == value
 
 
 def test_id_property_wrong_type():
@@ -139,17 +139,13 @@ def test_id_property_wrong_type():
 @pytest.mark.parametrize(
     "value", [
         'my-type--foo',
-        # Not a v4 UUID
+        # Not a RFC 4122 UUID
         'my-type--00000000-0000-0000-0000-000000000000',
-        'my-type--' + str(uuid.uuid1()),
-        'my-type--' + str(uuid.uuid3(uuid.NAMESPACE_DNS, "example.org")),
-        'my-type--' + str(uuid.uuid5(uuid.NAMESPACE_DNS, "example.org")),
     ],
 )
 def test_id_property_not_a_valid_hex_uuid(value):
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(ValueError):
         ID_PROP.clean(value)
-    assert str(excinfo.value) == ERROR_INVALID_ID
 
 
 def test_id_property_default():
@@ -275,17 +271,27 @@ def test_boolean_property_invalid(value):
 
 
 def test_reference_property():
-    ref_prop = ReferenceProperty()
+    ref_prop = ReferenceProperty(valid_types="my-type", spec_version="2.1")
 
     assert ref_prop.clean("my-type--00000000-0000-4000-8000-000000000000")
     with pytest.raises(ValueError):
         ref_prop.clean("foo")
 
-    # This is not a valid V4 UUID
+    # This is not a valid RFC 4122 UUID
     with pytest.raises(ValueError):
         ref_prop.clean("my-type--00000000-0000-0000-0000-000000000000")
 
 
+def test_reference_property_specific_type():
+    ref_prop = ReferenceProperty(valid_types="my-type", spec_version="2.1")
+
+    with pytest.raises(ValueError):
+        ref_prop.clean("not-my-type--8a8e8758-f92c-4058-ba38-f061cd42a0cf")
+
+    assert ref_prop.clean("my-type--8a8e8758-f92c-4058-ba38-f061cd42a0cf") == \
+        "my-type--8a8e8758-f92c-4058-ba38-f061cd42a0cf"
+
+
 @pytest.mark.parametrize(
     "value", [
         '2017-01-01T12:34:56Z',
@@ -470,23 +476,27 @@ def test_extension_property_valid():
     })
 
 
-@pytest.mark.parametrize(
-    "data", [
-        1,
-        {'foobar-ext': {
-            'pe_type': 'exe',
-        }},
-    ],
-)
-def test_extension_property_invalid(data):
+def test_extension_property_invalid1():
     ext_prop = ExtensionsProperty(spec_version='2.1', enclosing_type='file')
     with pytest.raises(ValueError):
-        ext_prop.clean(data)
+        ext_prop.clean(1)
+
+
+def test_extension_property_invalid2():
+    ext_prop = ExtensionsProperty(spec_version='2.1', enclosing_type='file')
+    with pytest.raises(CustomContentError):
+        ext_prop.clean(
+            {
+                'foobar-ext': {
+                    'pe_type': 'exe',
+                },
+            },
+        )
 
 
 def test_extension_property_invalid_type():
     ext_prop = ExtensionsProperty(spec_version='2.1', enclosing_type='indicator')
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(CustomContentError) as excinfo:
         ext_prop.clean(
             {
                 'windows-pebinary-ext': {

stix2/test/v21/test_report.py

@@ -91,8 +91,6 @@ def test_report_example_objects_in_object_refs_with_bad_id():
 
     assert excinfo.value.cls == stix2.v21.Report
     assert excinfo.value.prop_name == "object_refs"
-    assert excinfo.value.reason == stix2.properties.ERROR_INVALID_ID
-    assert str(excinfo.value) == "Invalid value for Report 'object_refs': " + stix2.properties.ERROR_INVALID_ID
 
 
 @pytest.mark.parametrize(

stix2/test/v21/test_sighting.py

@@ -61,8 +61,6 @@ def test_sighting_bad_where_sighted_refs():
 
     assert excinfo.value.cls == stix2.v21.Sighting
     assert excinfo.value.prop_name == "where_sighted_refs"
-    assert excinfo.value.reason == "must start with 'identity'."
-    assert str(excinfo.value) == "Invalid value for Sighting 'where_sighted_refs': must start with 'identity'."
 
 
 def test_sighting_type_must_be_sightings():

stix2/test/v21/test_threat_actor.py

@@ -4,6 +4,7 @@ import pytest
 import pytz
 
 import stix2
+import stix2.v21
 
 from .constants import IDENTITY_ID, THREAT_ACTOR_ID
 
@@ -67,4 +68,26 @@ def test_parse_threat_actor(data):
     assert actor.name == "Evil Org"
     assert actor.threat_actor_types == ["crime-syndicate"]
 
+
+def test_seen_ordering_constraint():
+    """
+    Test first_seen/last_seen value co-constraint.
+    """
+    with pytest.raises(ValueError):
+        stix2.v21.ThreatActor(
+            name="Bad Person",
+            threat_actor_types=["bad person", "evil person"],
+            first_seen="2010-04-21T09:31:11Z",
+            last_seen="2009-02-06T03:39:31Z",
+        )
+
+    # equal timestamps is okay.
+    stix2.v21.ThreatActor(
+        name="Bad Person",
+        threat_actor_types=["bad person", "evil person"],
+        first_seen="2010-04-21T09:31:11Z",
+        last_seen="2010-04-21T09:31:11Z",
+    )
+
+
 # TODO: Add other examples

stix2/test/v21/test_utils.py

@@ -135,14 +135,16 @@ def test_deduplicate(stix_objs1):
                     "0": {
                         "name": "foo.exe",
                         "type": "file",
+                        "id": "file--5956efbb-a7b0-566d-a7f9-a202eb05c70f",
                     },
                     "1": {
                         "type": "ipv4-addr",
                         "value": "198.51.100.3",
+                        "id": "ipv4-addr--1f8f4d63-9f33-5353-a3e3-e1b84c83a7b5",
                     },
                     "2": {
                         "type": "network-traffic",
-                        "src_ref": "1",
+                        "src_ref": "ipv4-addr--1f8f4d63-9f33-5353-a3e3-e1b84c83a7b5",
                         "protocols": [
                           "tcp",
                           "http",
@@ -161,7 +163,7 @@ def test_deduplicate(stix_objs1):
                         },
                     },
                 },
-            ), ('1', {"type": "ipv4-addr", "value": "198.51.100.3"}), 1,
+            ), ('1', {"type": "ipv4-addr", "value": "198.51.100.3", "id": "ipv4-addr--1f8f4d63-9f33-5353-a3e3-e1b84c83a7b5"}), 1,
         ),
         (
             {

stix2/test/v21/test_versioning.py

@@ -230,6 +230,7 @@ def test_remove_custom_stix_property():
         malware_types=["rootkit"],
         x_custom="armada",
         allow_custom=True,
+        is_family=False,
     )
 
     mal_nc = stix2.utils.remove_custom_stix(mal)

stix2/test/v21/test_workbench.py

@@ -1,331 +0,0 @@
-import os
-
-import pytest
-
-import stix2
-from stix2.workbench import (
-    AttackPattern, Campaign, CourseOfAction, ExternalReference,
-    FileSystemSource, Filter, Identity, Indicator, IntrusionSet, Malware,
-    MarkingDefinition, ObservedData, Relationship, Report, StatementMarking,
-    ThreatActor, Tool, Vulnerability, add_data_source, all_versions,
-    attack_patterns, campaigns, courses_of_action, create, get, identities,
-    indicators, intrusion_sets, malware, observed_data, query, reports, save,
-    set_default_created, set_default_creator, set_default_external_refs,
-    set_default_object_marking_refs, threat_actors, tools, vulnerabilities,
-)
-
-from .constants import (
-    ATTACK_PATTERN_ID, ATTACK_PATTERN_KWARGS, CAMPAIGN_ID, CAMPAIGN_KWARGS,
-    COURSE_OF_ACTION_ID, COURSE_OF_ACTION_KWARGS, IDENTITY_ID, IDENTITY_KWARGS,
-    INDICATOR_ID, INDICATOR_KWARGS, INTRUSION_SET_ID, INTRUSION_SET_KWARGS,
-    MALWARE_ID, MALWARE_KWARGS, OBSERVED_DATA_ID, OBSERVED_DATA_KWARGS,
-    REPORT_ID, REPORT_KWARGS, THREAT_ACTOR_ID, THREAT_ACTOR_KWARGS, TOOL_ID,
-    TOOL_KWARGS, VULNERABILITY_ID, VULNERABILITY_KWARGS,
-)
-
-
-@pytest.mark.xfail(reason='The workbench is not working correctly for 2.1')
-def test_workbench_environment():
-
-    # Create a STIX object
-    ind = create(Indicator, id=INDICATOR_ID, **INDICATOR_KWARGS)
-    save(ind)
-
-    resp = get(INDICATOR_ID)
-    assert resp['indicator_types'][0] == 'malicious-activity'
-
-    resp = all_versions(INDICATOR_ID)
-    assert len(resp) == 1
-
-    # Search on something other than id
-    q = [Filter('type', '=', 'vulnerability')]
-    resp = query(q)
-    assert len(resp) == 0
-
-
-def test_workbench_get_all_attack_patterns():
-    mal = AttackPattern(id=ATTACK_PATTERN_ID, **ATTACK_PATTERN_KWARGS)
-    save(mal)
-
-    resp = attack_patterns()
-    assert len(resp) == 1
-    assert resp[0].id == ATTACK_PATTERN_ID
-
-
-def test_workbench_get_all_campaigns():
-    cam = Campaign(id=CAMPAIGN_ID, **CAMPAIGN_KWARGS)
-    save(cam)
-
-    resp = campaigns()
-    assert len(resp) == 1
-    assert resp[0].id == CAMPAIGN_ID
-
-
-def test_workbench_get_all_courses_of_action():
-    coa = CourseOfAction(id=COURSE_OF_ACTION_ID, **COURSE_OF_ACTION_KWARGS)
-    save(coa)
-
-    resp = courses_of_action()
-    assert len(resp) == 1
-    assert resp[0].id == COURSE_OF_ACTION_ID
-
-
-def test_workbench_get_all_identities():
-    idty = Identity(id=IDENTITY_ID, **IDENTITY_KWARGS)
-    save(idty)
-
-    resp = identities()
-    assert len(resp) == 1
-    assert resp[0].id == IDENTITY_ID
-
-
-@pytest.mark.xfail(reason='The workbench is not working correctly for 2.1')
-def test_workbench_get_all_indicators():
-    resp = indicators()
-    assert len(resp) == 1
-    assert resp[0].id == INDICATOR_ID
-
-
-def test_workbench_get_all_intrusion_sets():
-    ins = IntrusionSet(id=INTRUSION_SET_ID, **INTRUSION_SET_KWARGS)
-    save(ins)
-
-    resp = intrusion_sets()
-    assert len(resp) == 1
-    assert resp[0].id == INTRUSION_SET_ID
-
-
-@pytest.mark.xfail(reason='The workbench is not working correctly for 2.1')
-def test_workbench_get_all_malware():
-    mal = Malware(id=MALWARE_ID, **MALWARE_KWARGS)
-    save(mal)
-
-    resp = malware()
-    assert len(resp) == 1
-    assert resp[0].id == MALWARE_ID
-
-
-def test_workbench_get_all_observed_data():
-    od = ObservedData(id=OBSERVED_DATA_ID, **OBSERVED_DATA_KWARGS)
-    save(od)
-
-    resp = observed_data()
-    assert len(resp) == 1
-    assert resp[0].id == OBSERVED_DATA_ID
-
-
-@pytest.mark.xfail(reason='The workbench is not working correctly for 2.1')
-def test_workbench_get_all_reports():
-    rep = Report(id=REPORT_ID, **REPORT_KWARGS)
-    save(rep)
-
-    resp = reports()
-    assert len(resp) == 1
-    assert resp[0].id == REPORT_ID
-
-
-@pytest.mark.xfail(reason='The workbench is not working correctly for 2.1')
-def test_workbench_get_all_threat_actors():
-    thr = ThreatActor(id=THREAT_ACTOR_ID, **THREAT_ACTOR_KWARGS)
-    save(thr)
-
-    resp = threat_actors()
-    assert len(resp) == 1
-    assert resp[0].id == THREAT_ACTOR_ID
-
-
-@pytest.mark.xfail(reason='The workbench is not working correctly for 2.1')
-def test_workbench_get_all_tools():
-    tool = Tool(id=TOOL_ID, **TOOL_KWARGS)
-    save(tool)
-
-    resp = tools()
-    assert len(resp) == 1
-    assert resp[0].id == TOOL_ID
-
-
-def test_workbench_get_all_vulnerabilities():
-    vuln = Vulnerability(id=VULNERABILITY_ID, **VULNERABILITY_KWARGS)
-    save(vuln)
-
-    resp = vulnerabilities()
-    assert len(resp) == 1
-    assert resp[0].id == VULNERABILITY_ID
-
-
-def test_workbench_add_to_bundle():
-    vuln = Vulnerability(**VULNERABILITY_KWARGS)
-    bundle = stix2.v21.Bundle(vuln)
-    assert bundle.objects[0].name == 'Heartbleed'
-
-
-@pytest.mark.xfail(reason='The workbench is not working correctly for 2.1')
-def test_workbench_relationships():
-    rel = Relationship(INDICATOR_ID, 'indicates', MALWARE_ID)
-    save(rel)
-
-    ind = get(INDICATOR_ID)
-    resp = ind.relationships()
-    assert len(resp) == 1
-    assert resp[0].relationship_type == 'indicates'
-    assert resp[0].source_ref == INDICATOR_ID
-    assert resp[0].target_ref == MALWARE_ID
-
-
-def test_workbench_created_by():
-    intset = IntrusionSet(name="Breach 123", created_by_ref=IDENTITY_ID)
-    save(intset)
-    creator = intset.created_by()
-    assert creator.id == IDENTITY_ID
-
-
-@pytest.mark.xfail(reason='The workbench is not working correctly for 2.1')
-def test_workbench_related():
-    rel1 = Relationship(MALWARE_ID, 'targets', IDENTITY_ID)
-    rel2 = Relationship(CAMPAIGN_ID, 'uses', MALWARE_ID)
-    save([rel1, rel2])
-
-    resp = get(MALWARE_ID).related()
-    assert len(resp) == 3
-    assert any(x['id'] == CAMPAIGN_ID for x in resp)
-    assert any(x['id'] == INDICATOR_ID for x in resp)
-    assert any(x['id'] == IDENTITY_ID for x in resp)
-
-    resp = get(MALWARE_ID).related(relationship_type='indicates')
-    assert len(resp) == 1
-
-
-@pytest.mark.xfail(reason='The workbench is not working correctly for 2.1')
-def test_workbench_related_with_filters():
-    malware = Malware(
-        malware_types=["ransomware"], name="CryptorBit",
-        created_by_ref=IDENTITY_ID,
-    )
-    rel = Relationship(malware.id, 'variant-of', MALWARE_ID)
-    save([malware, rel])
-
-    filters = [Filter('created_by_ref', '=', IDENTITY_ID)]
-    resp = get(MALWARE_ID).related(filters=filters)
-
-    assert len(resp) == 1
-    assert resp[0].name == malware.name
-    assert resp[0].created_by_ref == IDENTITY_ID
-
-    # filters arg can also be single filter
-    resp = get(MALWARE_ID).related(filters=filters[0])
-    assert len(resp) == 1
-
-
-@pytest.mark.xfail(reason='The workbench is not working correctly for 2.1')
-def test_add_data_source():
-    fs_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "stix2_data")
-    fs = FileSystemSource(fs_path)
-    add_data_source(fs)
-
-    resp = tools()
-    assert len(resp) == 3
-    resp_ids = [tool.id for tool in resp]
-    assert TOOL_ID in resp_ids
-    assert 'tool--03342581-f790-4f03-ba41-e82e67392e23' in resp_ids
-    assert 'tool--242f3da3-4425-4d11-8f5c-b842886da966' in resp_ids
-
-
-def test_additional_filter():
-    resp = tools(Filter('created_by_ref', '=', 'identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5'))
-    assert len(resp) == 2
-
-
-def test_additional_filters_list():
-    resp = tools([
-        Filter('created_by_ref', '=', 'identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5'),
-        Filter('name', '=', 'Windows Credential Editor'),
-    ])
-    assert len(resp) == 1
-
-
-def test_default_creator():
-    set_default_creator(IDENTITY_ID)
-    campaign = Campaign(**CAMPAIGN_KWARGS)
-
-    assert 'created_by_ref' not in CAMPAIGN_KWARGS
-    assert campaign.created_by_ref == IDENTITY_ID
-
-
-def test_default_created_timestamp():
-    timestamp = "2018-03-19T01:02:03.000Z"
-    set_default_created(timestamp)
-    campaign = Campaign(**CAMPAIGN_KWARGS)
-
-    assert 'created' not in CAMPAIGN_KWARGS
-    assert stix2.utils.format_datetime(campaign.created) == timestamp
-    assert stix2.utils.format_datetime(campaign.modified) == timestamp
-
-
-def test_default_external_refs():
-    ext_ref = ExternalReference(
-        source_name="ACME Threat Intel",
-        description="Threat report",
-    )
-    set_default_external_refs(ext_ref)
-    campaign = Campaign(**CAMPAIGN_KWARGS)
-
-    assert campaign.external_references[0].source_name == "ACME Threat Intel"
-    assert campaign.external_references[0].description == "Threat report"
-
-
-def test_default_object_marking_refs():
-    stmt_marking = StatementMarking("Copyright 2016, Example Corp")
-    mark_def = MarkingDefinition(
-        definition_type="statement",
-        definition=stmt_marking,
-    )
-    set_default_object_marking_refs(mark_def)
-    campaign = Campaign(**CAMPAIGN_KWARGS)
-
-    assert campaign.object_marking_refs[0] == mark_def.id
-
-
-def test_workbench_custom_property_object_in_observable_extension():
-    ntfs = stix2.v21.NTFSExt(
-        allow_custom=True,
-        sid=1,
-        x_foo='bar',
-    )
-    artifact = stix2.v21.File(
-        name='test',
-        extensions={'ntfs-ext': ntfs},
-    )
-    observed_data = ObservedData(
-        allow_custom=True,
-        first_observed="2015-12-21T19:00:00Z",
-        last_observed="2015-12-21T19:00:00Z",
-        number_observed=1,
-        objects={"0": artifact},
-    )
-
-    assert observed_data.objects['0'].extensions['ntfs-ext'].x_foo == "bar"
-    assert '"x_foo": "bar"' in str(observed_data)
-
-
-def test_workbench_custom_property_dict_in_observable_extension():
-    artifact = stix2.v21.File(
-        allow_custom=True,
-        name='test',
-        extensions={
-            'ntfs-ext': {
-                'allow_custom': True,
-                'sid': 1,
-                'x_foo': 'bar',
-            },
-        },
-    )
-    observed_data = ObservedData(
-        allow_custom=True,
-        first_observed="2015-12-21T19:00:00Z",
-        last_observed="2015-12-21T19:00:00Z",
-        number_observed=1,
-        objects={"0": artifact},
-    )
-
-    assert observed_data.objects['0'].extensions['ntfs-ext'].x_foo == "bar"
-    assert '"x_foo": "bar"' in str(observed_data)

stix2/v20/bundle.py

@@ -16,11 +16,11 @@ class Bundle(_STIXBase):
     _type = 'bundle'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.0')),
         # Not technically correct: STIX 2.0 spec doesn't say spec_version must
         # have this value, but it's all we support for now.
         ('spec_version', StringProperty(fixed='2.0')),
-        ('objects', ListProperty(STIXObjectProperty)),
+        ('objects', ListProperty(STIXObjectProperty(spec_version="2.0"))),
     ])
 
     def __init__(self, *args, **kwargs):

stix2/v20/common.py

@@ -3,6 +3,8 @@
 from collections import OrderedDict
 import copy
 
+import six
+
 from ..base import _STIXBase
 from ..custom import _custom_marking_builder
 from ..markings import _MarkingsMixin
@@ -14,6 +16,21 @@ from ..properties import (
 from ..utils import NOW, _get_dict
 
 
+def _should_set_millisecond(cr, marking_type):
+    # TLP instances in the 2.0 spec have millisecond precision unlike other markings
+    if marking_type == TLPMarking:
+        return True
+    # otherwise,  precision is kept from how it was given
+    if isinstance(cr, six.string_types):
+        if '.' in cr:
+            return True
+        else:
+            return False
+    if cr.precision == 'millisecond':
+        return True
+    return False
+
+
 class ExternalReference(_STIXBase):
     """For more detailed information on this object's properties, see
     `the STIX 2.0 specification <http://docs.oasis-open.org/cti/stix/v2.0/cs01/part1-stix-core/stix-v2.0-cs01-part1-stix-core.html#_Toc496709261>`__.
@@ -49,7 +66,7 @@ class GranularMarking(_STIXBase):
     """
 
     _properties = OrderedDict([
-        ('marking_ref', ReferenceProperty(required=True, type='marking-definition')),
+        ('marking_ref', ReferenceProperty(valid_types='marking-definition', spec_version='2.0', required=True)),
         ('selectors', ListProperty(SelectorProperty, required=True)),
     ])
 
@@ -104,11 +121,11 @@ class MarkingDefinition(_STIXBase, _MarkingsMixin):
     _type = 'marking-definition'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.0')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.0')),
         ('created', TimestampProperty(default=lambda: NOW)),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.0'))),
         ('granular_markings', ListProperty(GranularMarking)),
         ('definition_type', StringProperty(required=True)),
         ('definition', MarkingProperty(required=True)),
@@ -122,12 +139,12 @@ class MarkingDefinition(_STIXBase, _MarkingsMixin):
             except KeyError:
                 raise ValueError("definition_type must be a valid marking type")
 
-            if marking_type == TLPMarking:
-                # TLP instances in the spec have millisecond precision unlike other markings
-                self._properties = copy.deepcopy(self._properties)
-                self._properties.update([
-                    ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
-                ])
+            if 'created' in kwargs:
+                if _should_set_millisecond(kwargs['created'], marking_type):
+                    self._properties = copy.deepcopy(self._properties)
+                    self._properties.update([
+                        ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
+                    ])
 
             if not isinstance(kwargs['definition'], marking_type):
                 defn = _get_dict(kwargs['definition'])

stix2/v20/observables.py

@@ -31,7 +31,7 @@ class Artifact(_Observable):
         ('payload_bin', BinaryProperty()),
         ('url', StringProperty()),
         ('hashes', HashesProperty()),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
     def _check_object_constraints(self):
@@ -51,7 +51,7 @@ class AutonomousSystem(_Observable):
         ('number', IntegerProperty(required=True)),
         ('name', StringProperty()),
         ('rir', StringProperty()),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
 
@@ -70,7 +70,7 @@ class Directory(_Observable):
         ('modified', TimestampProperty()),
         ('accessed', TimestampProperty()),
         ('contains_refs', ListProperty(ObjectReferenceProperty(valid_types=['file', 'directory']))),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
 
@@ -84,7 +84,7 @@ class DomainName(_Observable):
         ('type', TypeProperty(_type)),
         ('value', StringProperty(required=True)),
         ('resolves_to_refs', ListProperty(ObjectReferenceProperty(valid_types=['ipv4-addr', 'ipv6-addr', 'domain-name']))),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
 
@@ -99,7 +99,7 @@ class EmailAddress(_Observable):
         ('value', StringProperty(required=True)),
         ('display_name', StringProperty()),
         ('belongs_to_ref', ObjectReferenceProperty(valid_types='user-account')),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
 
@@ -138,11 +138,11 @@ class EmailMessage(_Observable):
         ('bcc_refs', ListProperty(ObjectReferenceProperty(valid_types='email-addr'))),
         ('subject', StringProperty()),
         ('received_lines', ListProperty(StringProperty)),
-        ('additional_header_fields', DictionaryProperty()),
+        ('additional_header_fields', DictionaryProperty(spec_version="2.0")),
         ('body', StringProperty()),
         ('body_multipart', ListProperty(EmbeddedObjectProperty(type=EmailMIMEComponent))),
         ('raw_email_ref', ObjectReferenceProperty(valid_types='artifact')),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
     def _check_object_constraints(self):
@@ -199,7 +199,7 @@ class PDFExt(_Extension):
     _properties = OrderedDict([
         ('version', StringProperty()),
         ('is_optimized', BooleanProperty()),
-        ('document_info_dict', DictionaryProperty()),
+        ('document_info_dict', DictionaryProperty(spec_version="2.0")),
         ('pdfid0', StringProperty()),
         ('pdfid1', StringProperty()),
     ])
@@ -216,7 +216,7 @@ class RasterImageExt(_Extension):
         ('image_width', IntegerProperty()),
         ('bits_per_pixel', IntegerProperty()),
         ('image_compression_algorithm', StringProperty()),
-        ('exif_tags', DictionaryProperty()),
+        ('exif_tags', DictionaryProperty(spec_version="2.0")),
     ])
 
 
@@ -323,7 +323,7 @@ class File(_Observable):
         ('decryption_key', StringProperty()),
         ('contains_refs', ListProperty(ObjectReferenceProperty)),
         ('content_ref', ObjectReferenceProperty(valid_types='artifact')),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
     def _check_object_constraints(self):
@@ -343,7 +343,7 @@ class IPv4Address(_Observable):
         ('value', StringProperty(required=True)),
         ('resolves_to_refs', ListProperty(ObjectReferenceProperty(valid_types='mac-addr'))),
         ('belongs_to_refs', ListProperty(ObjectReferenceProperty(valid_types='autonomous-system'))),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
 
@@ -358,7 +358,7 @@ class IPv6Address(_Observable):
         ('value', StringProperty(required=True)),
         ('resolves_to_refs', ListProperty(ObjectReferenceProperty(valid_types='mac-addr'))),
         ('belongs_to_refs', ListProperty(ObjectReferenceProperty(valid_types='autonomous-system'))),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
 
@@ -371,7 +371,7 @@ class MACAddress(_Observable):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('value', StringProperty(required=True)),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
 
@@ -384,7 +384,7 @@ class Mutex(_Observable):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('name', StringProperty(required=True)),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
 
@@ -398,7 +398,7 @@ class HTTPRequestExt(_Extension):
         ('request_method', StringProperty(required=True)),
         ('request_value', StringProperty(required=True)),
         ('request_version', StringProperty()),
-        ('request_header', DictionaryProperty()),
+        ('request_header', DictionaryProperty(spec_version="2.0")),
         ('message_body_length', IntegerProperty()),
         ('message_body_data_ref', ObjectReferenceProperty(valid_types='artifact')),
     ])
@@ -449,7 +449,7 @@ class SocketExt(_Extension):
                 "PF_NETROM",
             ]),
         ),
-        ('options', DictionaryProperty()),
+        ('options', DictionaryProperty(spec_version="2.0")),
         (
             'socket_type', EnumProperty(allowed=[
                 "SOCK_STREAM",
@@ -496,12 +496,12 @@ class NetworkTraffic(_Observable):
         ('dst_byte_count', IntegerProperty()),
         ('src_packets', IntegerProperty()),
         ('dst_packets', IntegerProperty()),
-        ('ipfix', DictionaryProperty()),
+        ('ipfix', DictionaryProperty(spec_version="2.0")),
         ('src_payload_ref', ObjectReferenceProperty(valid_types='artifact')),
         ('dst_payload_ref', ObjectReferenceProperty(valid_types='artifact')),
         ('encapsulates_refs', ListProperty(ObjectReferenceProperty(valid_types='network-traffic'))),
         ('encapsulates_by_ref', ObjectReferenceProperty(valid_types='network-traffic')),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
     def _check_object_constraints(self):
@@ -521,7 +521,7 @@ class WindowsProcessExt(_Extension):
         ('priority', StringProperty()),
         ('owner_sid', StringProperty()),
         ('window_title', StringProperty()),
-        ('startup_info', DictionaryProperty()),
+        ('startup_info', DictionaryProperty(spec_version="2.0")),
     ])
 
 
@@ -584,13 +584,13 @@ class Process(_Observable):
         ('cwd', StringProperty()),
         ('arguments', ListProperty(StringProperty)),
         ('command_line', StringProperty()),
-        ('environment_variables', DictionaryProperty()),
+        ('environment_variables', DictionaryProperty(spec_version="2.0")),
         ('opened_connection_refs', ListProperty(ObjectReferenceProperty(valid_types='network-traffic'))),
         ('creator_user_ref', ObjectReferenceProperty(valid_types='user-account')),
         ('binary_ref', ObjectReferenceProperty(valid_types='file')),
         ('parent_ref', ObjectReferenceProperty(valid_types='process')),
         ('child_refs', ListProperty(ObjectReferenceProperty('process'))),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
     def _check_object_constraints(self):
@@ -621,7 +621,7 @@ class Software(_Observable):
         ('languages', ListProperty(StringProperty)),
         ('vendor', StringProperty()),
         ('version', StringProperty()),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
 
@@ -634,7 +634,7 @@ class URL(_Observable):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('value', StringProperty(required=True)),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
 
@@ -673,7 +673,7 @@ class UserAccount(_Observable):
         ('password_last_changed', TimestampProperty()),
         ('account_first_login', TimestampProperty()),
         ('account_last_login', TimestampProperty()),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
 
@@ -720,7 +720,7 @@ class WindowsRegistryKey(_Observable):
         ('modified', TimestampProperty()),
         ('creator_user_ref', ObjectReferenceProperty(valid_types='user-account')),
         ('number_of_subkeys', IntegerProperty()),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
     @property
@@ -776,7 +776,7 @@ class X509Certificate(_Observable):
         ('subject_public_key_modulus', StringProperty()),
         ('subject_public_key_exponent', IntegerProperty()),
         ('x509_v3_extensions', EmbeddedObjectProperty(type=X509V3ExtenstionsType)),
-        ('extensions', ExtensionsProperty(enclosing_type=_type)),
+        ('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=_type)),
     ])
 
 
@@ -798,7 +798,7 @@ def CustomObservable(type='x-custom-observable', properties=None):
         _properties = list(itertools.chain.from_iterable([
             [('type', TypeProperty(type))],
             properties,
-            [('extensions', ExtensionsProperty(enclosing_type=type))],
+            [('extensions', ExtensionsProperty(spec_version="2.0", enclosing_type=type))],
         ]))
         return _custom_observable_builder(cls, type, _properties, '2.0')
     return wrapper

stix2/v20/sdo.py

@@ -22,8 +22,8 @@ class AttackPattern(STIXDomainObject):
     _type = 'attack-pattern'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.0')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.0')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
@@ -32,7 +32,7 @@ class AttackPattern(STIXDomainObject):
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.0'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -45,8 +45,8 @@ class Campaign(STIXDomainObject):
     _type = 'campaign'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.0')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.0')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
@@ -58,7 +58,7 @@ class Campaign(STIXDomainObject):
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.0'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -71,8 +71,8 @@ class CourseOfAction(STIXDomainObject):
     _type = 'course-of-action'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.0')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.0')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
@@ -80,7 +80,7 @@ class CourseOfAction(STIXDomainObject):
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.0'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -93,8 +93,8 @@ class Identity(STIXDomainObject):
     _type = 'identity'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.0')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.0')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
@@ -105,7 +105,7 @@ class Identity(STIXDomainObject):
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.0'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -118,8 +118,8 @@ class Indicator(STIXDomainObject):
     _type = 'indicator'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.0')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.0')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty()),
@@ -131,7 +131,7 @@ class Indicator(STIXDomainObject):
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty, required=True)),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.0'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -144,8 +144,8 @@ class IntrusionSet(STIXDomainObject):
     _type = 'intrusion-set'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.0')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.0')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
@@ -160,7 +160,7 @@ class IntrusionSet(STIXDomainObject):
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.0'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -173,8 +173,8 @@ class Malware(STIXDomainObject):
     _type = 'malware'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.0')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.0')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
@@ -183,7 +183,7 @@ class Malware(STIXDomainObject):
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty, required=True)),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.0'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -196,18 +196,18 @@ class ObservedData(STIXDomainObject):
     _type = 'observed-data'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.0')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.0')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('first_observed', TimestampProperty(required=True)),
         ('last_observed', TimestampProperty(required=True)),
         ('number_observed', IntegerProperty(min=1, max=999999999, required=True)),
-        ('objects', ObservableProperty(required=True)),
+        ('objects', ObservableProperty(spec_version="2.0", required=True)),
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.0'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -226,18 +226,18 @@ class Report(STIXDomainObject):
     _type = 'report'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.0')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.0')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
         ('description', StringProperty()),
         ('published', TimestampProperty(required=True)),
-        ('object_refs', ListProperty(ReferenceProperty, required=True)),
+        ('object_refs', ListProperty(ReferenceProperty(invalid_types=[""], spec_version='2.0'), required=True)),
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty, required=True)),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.0'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -255,8 +255,8 @@ class ThreatActor(STIXDomainObject):
     _type = 'threat-actor'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.0')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.0')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
@@ -272,7 +272,7 @@ class ThreatActor(STIXDomainObject):
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty, required=True)),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.0'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -285,8 +285,8 @@ class Tool(STIXDomainObject):
     _type = 'tool'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.0')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.0')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
@@ -296,7 +296,7 @@ class Tool(STIXDomainObject):
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty, required=True)),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.0'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -309,8 +309,8 @@ class Vulnerability(STIXDomainObject):
     _type = 'vulnerability'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.0')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.0')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
@@ -318,7 +318,7 @@ class Vulnerability(STIXDomainObject):
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.0'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -356,8 +356,8 @@ def CustomObject(type='x-custom-type', properties=None):
         _properties = list(itertools.chain.from_iterable([
             [
                 ('type', TypeProperty(type)),
-                ('id', IDProperty(type)),
-                ('created_by_ref', ReferenceProperty(type='identity')),
+                ('id', IDProperty(type, spec_version='2.0')),
+                ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.0')),
                 ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
                 ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
             ],
@@ -366,7 +366,7 @@ def CustomObject(type='x-custom-type', properties=None):
                 ('revoked', BooleanProperty(default=lambda: False)),
                 ('labels', ListProperty(StringProperty)),
                 ('external_references', ListProperty(ExternalReference)),
-                ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+                ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.0'))),
                 ('granular_markings', ListProperty(GranularMarking)),
             ],
             sorted([x for x in properties if x[0].startswith('x_')], key=lambda x: x[0]),

stix2/v20/sro.py

@@ -16,21 +16,23 @@ class Relationship(STIXRelationshipObject):
     `the STIX 2.0 specification <http://docs.oasis-open.org/cti/stix/v2.0/cs01/part2-stix-objects/stix-v2.0-cs01-part2-stix-objects.html#_Toc496714340>`__.
     """
 
+    _invalid_source_target_types = ['bundle', 'language-content', 'marking-definition', 'relationship', 'sighting']
+
     _type = 'relationship'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.0')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.0')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('relationship_type', StringProperty(required=True)),
         ('description', StringProperty()),
-        ('source_ref', ReferenceProperty(required=True)),
-        ('target_ref', ReferenceProperty(required=True)),
+        ('source_ref', ReferenceProperty(invalid_types=_invalid_source_target_types, spec_version='2.0', required=True)),
+        ('target_ref', ReferenceProperty(invalid_types=_invalid_source_target_types, spec_version='2.0', required=True)),
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.0'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -61,21 +63,21 @@ class Sighting(STIXRelationshipObject):
     _type = 'sighting'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.0')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.0')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('first_seen', TimestampProperty()),
         ('last_seen', TimestampProperty()),
         ('count', IntegerProperty(min=0, max=999999999)),
-        ('sighting_of_ref', ReferenceProperty(required=True)),
-        ('observed_data_refs', ListProperty(ReferenceProperty(type='observed-data'))),
-        ('where_sighted_refs', ListProperty(ReferenceProperty(type='identity'))),
+        ('sighting_of_ref', ReferenceProperty(valid_types="only_SDO", spec_version='2.0', required=True)),
+        ('observed_data_refs', ListProperty(ReferenceProperty(valid_types='observed-data', spec_version='2.0'))),
+        ('where_sighted_refs', ListProperty(ReferenceProperty(valid_types='identity', spec_version='2.0'))),
         ('summary', BooleanProperty(default=lambda: False)),
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.0'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 

stix2/v21/__init__.py

@@ -32,9 +32,10 @@ from .observables import (
     X509Certificate, X509V3ExtenstionsType,
 )
 from .sdo import (
-    AttackPattern, Campaign, CourseOfAction, CustomObject, Identity, Indicator,
-    IntrusionSet, Location, Malware, Note, ObservedData, Opinion, Report,
-    ThreatActor, Tool, Vulnerability,
+    AttackPattern, Campaign, CourseOfAction, CustomObject, Grouping, Identity,
+    Indicator, Infrastructure, IntrusionSet, Location, Malware,
+    MalwareAnalysis, Note, ObservedData, Opinion, Report, ThreatActor, Tool,
+    Vulnerability,
 )
 from .sro import Relationship, Sighting
 
@@ -43,12 +44,15 @@ OBJ_MAP = {
     'bundle': Bundle,
     'campaign': Campaign,
     'course-of-action': CourseOfAction,
+    'grouping': Grouping,
     'identity': Identity,
     'indicator': Indicator,
+    'infrastructure': Infrastructure,
     'intrusion-set': IntrusionSet,
     'language-content': LanguageContent,
     'location': Location,
     'malware': Malware,
+    'malware-analysis': MalwareAnalysis,
     'note': Note,
     'marking-definition': MarkingDefinition,
     'observed-data': ObservedData,

stix2/v21/bundle.py

@@ -17,7 +17,7 @@ class Bundle(_STIXBase):
     _type = 'bundle'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
-        ('id', IDProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('objects', ListProperty(STIXObjectProperty(spec_version='2.1'))),
     ])
 

stix2/v21/common.py

@@ -1,7 +1,6 @@
 """STIX 2.1 Common Data Types and Properties."""
 
 from collections import OrderedDict
-import copy
 
 from ..base import _STIXBase
 from ..custom import _custom_marking_builder
@@ -54,7 +53,7 @@ class GranularMarking(_STIXBase):
 
     _properties = OrderedDict([
         ('lang', StringProperty()),
-        ('marking_ref', ReferenceProperty(type='marking-definition')),
+        ('marking_ref', ReferenceProperty(valid_types='marking-definition', spec_version='2.1')),
         ('selectors', ListProperty(SelectorProperty, required=True)),
     ])
 
@@ -73,20 +72,20 @@ class LanguageContent(_STIXBase):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
-        ('object_ref', ReferenceProperty(required=True)),
+        ('object_ref', ReferenceProperty(invalid_types=[""], spec_version='2.1', required=True)),
         # TODO: 'object_modified' it MUST be an exact match for the modified time of the STIX Object (SRO or SDO) being referenced.
-        ('object_modified', TimestampProperty(required=True, precision='millisecond')),
+        ('object_modified', TimestampProperty(precision='millisecond')),
         # TODO: 'contents' https://docs.google.com/document/d/1ShNq4c3e1CkfANmD9O--mdZ5H0O_GLnjN28a_yrEaco/edit#heading=h.cfz5hcantmvx
         ('contents', DictionaryProperty(spec_version='2.1', required=True)),
         ('revoked', BooleanProperty()),
         ('labels', ListProperty(StringProperty)),
         ('confidence', IntegerProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -145,10 +144,10 @@ class MarkingDefinition(_STIXBase, _MarkingsMixin):
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
         ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
-        ('created', TimestampProperty(default=lambda: NOW)),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
+        ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
         ('definition_type', StringProperty(required=True)),
         ('definition', MarkingProperty(required=True)),
@@ -162,13 +161,6 @@ class MarkingDefinition(_STIXBase, _MarkingsMixin):
             except KeyError:
                 raise ValueError("definition_type must be a valid marking type")
 
-            if marking_type == TLPMarking:
-                # TLP instances in the spec have millisecond precision unlike other markings
-                self._properties = copy.deepcopy(self._properties)
-                self._properties.update([
-                    ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
-                ])
-
             if not isinstance(kwargs['definition'], marking_type):
                 defn = _get_dict(kwargs['definition'])
                 kwargs['definition'] = marking_type(**defn)

stix2/v21/observables.py

@@ -7,16 +7,21 @@ Observable and do not have a ``_type`` attribute.
 
 from collections import OrderedDict
 import itertools
+import warnings
 
 from ..base import _Extension, _Observable, _STIXBase
 from ..custom import _custom_extension_builder, _custom_observable_builder
-from ..exceptions import AtLeastOnePropertyError, DependentPropertiesError
+from ..exceptions import (
+    AtLeastOnePropertyError, DependentPropertiesError, STIXDeprecationWarning,
+)
 from ..properties import (
     BinaryProperty, BooleanProperty, CallableValues, DictionaryProperty,
     EmbeddedObjectProperty, EnumProperty, ExtensionsProperty, FloatProperty,
-    HashesProperty, HexProperty, IntegerProperty, ListProperty,
-    ObjectReferenceProperty, StringProperty, TimestampProperty, TypeProperty,
+    HashesProperty, HexProperty, IDProperty, IntegerProperty, ListProperty,
+    ObjectReferenceProperty, ReferenceProperty, StringProperty,
+    TimestampProperty, TypeProperty,
 )
+from .common import GranularMarking
 
 
 class Artifact(_Observable):
@@ -28,6 +33,7 @@ class Artifact(_Observable):
     _type = 'artifact'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('mime_type', StringProperty()),
         ('payload_bin', BinaryProperty()),
         ('url', StringProperty()),
@@ -35,7 +41,12 @@ class Artifact(_Observable):
         ('encryption_algorithm', StringProperty()),
         ('decryption_key', StringProperty()),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["hashes", "payload_bin"]
 
     def _check_object_constraints(self):
         super(Artifact, self)._check_object_constraints()
@@ -52,11 +63,17 @@ class AutonomousSystem(_Observable):
     _type = 'autonomous-system'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('number', IntegerProperty(required=True)),
         ('name', StringProperty()),
         ('rir', StringProperty()),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["number"]
 
 
 class Directory(_Observable):
@@ -68,15 +85,21 @@ class Directory(_Observable):
     _type = 'directory'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('path', StringProperty(required=True)),
         ('path_enc', StringProperty()),
         # these are not the created/modified timestamps of the object itself
-        ('created', TimestampProperty()),
-        ('modified', TimestampProperty()),
-        ('accessed', TimestampProperty()),
-        ('contains_refs', ListProperty(ObjectReferenceProperty(valid_types=['file', 'directory']))),
+        ('ctime', TimestampProperty()),
+        ('mtime', TimestampProperty()),
+        ('atime', TimestampProperty()),
+        ('contains_refs', ListProperty(ReferenceProperty(valid_types=['file', 'directory'], spec_version='2.1'))),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["path"]
 
 
 class DomainName(_Observable):
@@ -88,10 +111,24 @@ class DomainName(_Observable):
     _type = 'domain-name'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('value', StringProperty(required=True)),
-        ('resolves_to_refs', ListProperty(ObjectReferenceProperty(valid_types=['ipv4-addr', 'ipv6-addr', 'domain-name']))),
+        ('resolves_to_refs', ListProperty(ReferenceProperty(valid_types=['ipv4-addr', 'ipv6-addr', 'domain-name'], spec_version='2.1'))),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["value"]
+
+    def _check_object_constraints(self):
+        if self.get('resolves_to_refs'):
+            warnings.warn(
+                "The 'resolves_to_refs' property of domain-name is deprecated in "
+                "STIX 2.1. Use the 'resolves-to' relationship type instead",
+                STIXDeprecationWarning,
+            )
 
 
 class EmailAddress(_Observable):
@@ -103,11 +140,17 @@ class EmailAddress(_Observable):
     _type = 'email-addr'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('value', StringProperty(required=True)),
         ('display_name', StringProperty()),
-        ('belongs_to_ref', ObjectReferenceProperty(valid_types='user-account')),
+        ('belongs_to_ref', ReferenceProperty(valid_types='user-account', spec_version='2.1')),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["value"]
 
 
 class EmailMIMEComponent(_STIXBase):
@@ -137,22 +180,29 @@ class EmailMessage(_Observable):
     _type = 'email-message'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('is_multipart', BooleanProperty(required=True)),
         ('date', TimestampProperty()),
         ('content_type', StringProperty()),
-        ('from_ref', ObjectReferenceProperty(valid_types='email-addr')),
-        ('sender_ref', ObjectReferenceProperty(valid_types='email-addr')),
-        ('to_refs', ListProperty(ObjectReferenceProperty(valid_types='email-addr'))),
-        ('cc_refs', ListProperty(ObjectReferenceProperty(valid_types='email-addr'))),
-        ('bcc_refs', ListProperty(ObjectReferenceProperty(valid_types='email-addr'))),
+        ('from_ref', ReferenceProperty(valid_types='email-addr', spec_version='2.1')),
+        ('sender_ref', ReferenceProperty(valid_types='email-addr', spec_version='2.1')),
+        ('to_refs', ListProperty(ReferenceProperty(valid_types='email-addr', spec_version='2.1'))),
+        ('cc_refs', ListProperty(ReferenceProperty(valid_types='email-addr', spec_version='2.1'))),
+        ('bcc_refs', ListProperty(ReferenceProperty(valid_types='email-addr', spec_version='2.1'))),
+        ('message_id', StringProperty()),
         ('subject', StringProperty()),
         ('received_lines', ListProperty(StringProperty)),
         ('additional_header_fields', DictionaryProperty(spec_version='2.1')),
         ('body', StringProperty()),
         ('body_multipart', ListProperty(EmbeddedObjectProperty(type=EmailMIMEComponent))),
-        ('raw_email_ref', ObjectReferenceProperty(valid_types='artifact')),
+        ('raw_email_ref', ReferenceProperty(valid_types='artifact', spec_version='2.1')),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["from_ref", "subject", "body"]
 
     def _check_object_constraints(self):
         super(EmailMessage, self)._check_object_constraints()
@@ -170,7 +220,7 @@ class ArchiveExt(_Extension):
 
     _type = 'archive-ext'
     _properties = OrderedDict([
-        ('contains_refs', ListProperty(ObjectReferenceProperty(valid_types='file'), required=True)),
+        ('contains_refs', ListProperty(ObjectReferenceProperty(valid_types=['file', 'directory']), required=True)),
         ('comment', StringProperty()),
     ])
 
@@ -323,6 +373,7 @@ class File(_Observable):
     _type = 'file'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('hashes', HashesProperty(spec_version='2.1')),
         ('size', IntegerProperty(min=0)),
         ('name', StringProperty()),
@@ -330,14 +381,19 @@ class File(_Observable):
         ('magic_number_hex', HexProperty()),
         ('mime_type', StringProperty()),
         # these are not the created/modified timestamps of the object itself
-        ('created', TimestampProperty()),
-        ('modified', TimestampProperty()),
-        ('accessed', TimestampProperty()),
-        ('parent_directory_ref', ObjectReferenceProperty(valid_types='directory')),
-        ('contains_refs', ListProperty(ObjectReferenceProperty)),
-        ('content_ref', ObjectReferenceProperty(valid_types='artifact')),
+        ('ctime', TimestampProperty()),
+        ('mtime', TimestampProperty()),
+        ('atime', TimestampProperty()),
+        ('parent_directory_ref', ReferenceProperty(valid_types='directory', spec_version='2.1')),
+        ('contains_refs', ListProperty(ReferenceProperty(invalid_types="", spec_version='2.1'))),
+        ('content_ref', ReferenceProperty(valid_types='artifact', spec_version='2.1')),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["hashes", "name", "extensions"]
 
     def _check_object_constraints(self):
         super(File, self)._check_object_constraints()
@@ -353,11 +409,32 @@ class IPv4Address(_Observable):
     _type = 'ipv4-addr'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('value', StringProperty(required=True)),
-        ('resolves_to_refs', ListProperty(ObjectReferenceProperty(valid_types='mac-addr'))),
-        ('belongs_to_refs', ListProperty(ObjectReferenceProperty(valid_types='autonomous-system'))),
+        ('resolves_to_refs', ListProperty(ReferenceProperty(valid_types='mac-addr', spec_version='2.1'))),
+        ('belongs_to_refs', ListProperty(ReferenceProperty(valid_types='autonomous-system', spec_version='2.1'))),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["value"]
+
+    def _check_object_constraints(self):
+        if self.get('resolves_to_refs'):
+            warnings.warn(
+                "The 'resolves_to_refs' property of ipv4-addr is deprecated in "
+                "STIX 2.1. Use the 'resolves-to' relationship type instead",
+                STIXDeprecationWarning,
+            )
+
+        if self.get('belongs_to_refs'):
+            warnings.warn(
+                "The 'belongs_to_refs' property of ipv4-addr is deprecated in "
+                "STIX 2.1. Use the 'belongs-to' relationship type instead",
+                STIXDeprecationWarning,
+            )
 
 
 class IPv6Address(_Observable):
@@ -369,11 +446,32 @@ class IPv6Address(_Observable):
     _type = 'ipv6-addr'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('value', StringProperty(required=True)),
-        ('resolves_to_refs', ListProperty(ObjectReferenceProperty(valid_types='mac-addr'))),
-        ('belongs_to_refs', ListProperty(ObjectReferenceProperty(valid_types='autonomous-system'))),
+        ('resolves_to_refs', ListProperty(ReferenceProperty(valid_types='mac-addr', spec_version='2.1'))),
+        ('belongs_to_refs', ListProperty(ReferenceProperty(valid_types='autonomous-system', spec_version='2.1'))),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["value"]
+
+    def _check_object_constraints(self):
+        if self.get('resolves_to_refs'):
+            warnings.warn(
+                "The 'resolves_to_refs' property of ipv6-addr is deprecated in "
+                "STIX 2.1. Use the 'resolves-to' relationship type instead",
+                STIXDeprecationWarning,
+            )
+
+        if self.get('belongs_to_refs'):
+            warnings.warn(
+                "The 'belongs_to_refs' property of ipv6-addr is deprecated in "
+                "STIX 2.1. Use the 'belongs-to' relationship type instead",
+                STIXDeprecationWarning,
+            )
 
 
 class MACAddress(_Observable):
@@ -385,9 +483,15 @@ class MACAddress(_Observable):
     _type = 'mac-addr'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('value', StringProperty(required=True)),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["value"]
 
 
 class Mutex(_Observable):
@@ -399,9 +503,15 @@ class Mutex(_Observable):
     _type = 'mutex'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('name', StringProperty(required=True)),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["name"]
 
 
 class HTTPRequestExt(_Extension):
@@ -505,11 +615,12 @@ class NetworkTraffic(_Observable):
     _type = 'network-traffic'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('start', TimestampProperty()),
         ('end', TimestampProperty()),
         ('is_active', BooleanProperty()),
-        ('src_ref', ObjectReferenceProperty(valid_types=['ipv4-addr', 'ipv6-addr', 'mac-addr', 'domain-name'])),
-        ('dst_ref', ObjectReferenceProperty(valid_types=['ipv4-addr', 'ipv6-addr', 'mac-addr', 'domain-name'])),
+        ('src_ref', ReferenceProperty(valid_types=['ipv4-addr', 'ipv6-addr', 'mac-addr', 'domain-name'], spec_version='2.1')),
+        ('dst_ref', ReferenceProperty(valid_types=['ipv4-addr', 'ipv6-addr', 'mac-addr', 'domain-name'], spec_version='2.1')),
         ('src_port', IntegerProperty(min=0, max=65535)),
         ('dst_port', IntegerProperty(min=0, max=65535)),
         ('protocols', ListProperty(StringProperty, required=True)),
@@ -518,12 +629,17 @@ class NetworkTraffic(_Observable):
         ('src_packets', IntegerProperty(min=0)),
         ('dst_packets', IntegerProperty(min=0)),
         ('ipfix', DictionaryProperty(spec_version='2.1')),
-        ('src_payload_ref', ObjectReferenceProperty(valid_types='artifact')),
-        ('dst_payload_ref', ObjectReferenceProperty(valid_types='artifact')),
-        ('encapsulates_refs', ListProperty(ObjectReferenceProperty(valid_types='network-traffic'))),
-        ('encapsulates_by_ref', ObjectReferenceProperty(valid_types='network-traffic')),
+        ('src_payload_ref', ReferenceProperty(valid_types='artifact', spec_version='2.1')),
+        ('dst_payload_ref', ReferenceProperty(valid_types='artifact', spec_version='2.1')),
+        ('encapsulates_refs', ListProperty(ReferenceProperty(valid_types='network-traffic', spec_version='2.1'))),
+        ('encapsulated_by_ref', ReferenceProperty(valid_types='network-traffic', spec_version='2.1')),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["start", "src_ref", "dst_ref", "src_port", "dst_port", "protocols"]
 
     def _check_object_constraints(self):
         super(NetworkTraffic, self)._check_object_constraints()
@@ -624,20 +740,26 @@ class Process(_Observable):
     _type = 'process'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('is_hidden', BooleanProperty()),
         ('pid', IntegerProperty()),
         # this is not the created timestamps of the object itself
-        ('created', TimestampProperty()),
+        ('created_time', TimestampProperty()),
         ('cwd', StringProperty()),
         ('command_line', StringProperty()),
         ('environment_variables', DictionaryProperty(spec_version='2.1')),
-        ('opened_connection_refs', ListProperty(ObjectReferenceProperty(valid_types='network-traffic'))),
-        ('creator_user_ref', ObjectReferenceProperty(valid_types='user-account')),
-        ('image_ref', ObjectReferenceProperty(valid_types='file')),
-        ('parent_ref', ObjectReferenceProperty(valid_types='process')),
-        ('child_refs', ListProperty(ObjectReferenceProperty('process'))),
+        ('opened_connection_refs', ListProperty(ReferenceProperty(valid_types='network-traffic', spec_version='2.1'))),
+        ('creator_user_ref', ReferenceProperty(valid_types='user-account', spec_version='2.1')),
+        ('image_ref', ReferenceProperty(valid_types='file', spec_version='2.1')),
+        ('parent_ref', ReferenceProperty(valid_types='process', spec_version='2.1')),
+        ('child_refs', ListProperty(ReferenceProperty(valid_types='process', spec_version='2.1'))),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = []
 
     def _check_object_constraints(self):
         # no need to check windows-service-ext, since it has a required property
@@ -663,13 +785,19 @@ class Software(_Observable):
     _type = 'software'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('name', StringProperty(required=True)),
         ('cpe', StringProperty()),
         ('languages', ListProperty(StringProperty)),
         ('vendor', StringProperty()),
         ('version', StringProperty()),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["name", "cpe", "vendor", "version"]
 
 
 class URL(_Observable):
@@ -681,9 +809,15 @@ class URL(_Observable):
     _type = 'url'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('value', StringProperty(required=True)),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["value"]
 
 
 class UNIXAccountExt(_Extension):
@@ -710,6 +844,7 @@ class UserAccount(_Observable):
     _type = 'user-account'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('user_id', StringProperty()),
         ('credential', StringProperty()),
         ('account_login', StringProperty()),
@@ -725,7 +860,12 @@ class UserAccount(_Observable):
         ('account_first_login', TimestampProperty()),
         ('account_last_login', TimestampProperty()),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["account_type", "user_id", "account_login"]
 
 
 class WindowsRegistryValueType(_STIXBase):
@@ -767,14 +907,20 @@ class WindowsRegistryKey(_Observable):
     _type = 'windows-registry-key'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('key', StringProperty()),
         ('values', ListProperty(EmbeddedObjectProperty(type=WindowsRegistryValueType))),
         # this is not the modified timestamps of the object itself
-        ('modified', TimestampProperty()),
-        ('creator_user_ref', ObjectReferenceProperty(valid_types='user-account')),
+        ('modified_time', TimestampProperty()),
+        ('creator_user_ref', ReferenceProperty(valid_types='user-account', spec_version='2.1')),
         ('number_of_subkeys', IntegerProperty()),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["key", "values"]
 
     @property
     def values(self):
@@ -818,6 +964,7 @@ class X509Certificate(_Observable):
     _type = 'x509-certificate'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
+        ('id', IDProperty(_type, spec_version='2.1')),
         ('is_self_signed', BooleanProperty()),
         ('hashes', HashesProperty(spec_version='2.1')),
         ('version', StringProperty()),
@@ -832,7 +979,12 @@ class X509Certificate(_Observable):
         ('subject_public_key_exponent', IntegerProperty()),
         ('x509_v3_extensions', EmbeddedObjectProperty(type=X509V3ExtenstionsType)),
         ('extensions', ExtensionsProperty(spec_version='2.1', enclosing_type=_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('defanged', BooleanProperty(default=lambda: False)),
     ])
+    _id_contributing_properties = ["hashes", "serial_number"]
 
 
 def CustomObservable(type='x-custom-observable', properties=None):

stix2/v21/sdo.py

@@ -2,15 +2,18 @@
 
 from collections import OrderedDict
 import itertools
+import warnings
 
 from six.moves.urllib.parse import quote_plus
 
 from ..core import STIXDomainObject
 from ..custom import _custom_object_builder
+from ..exceptions import PropertyPresenceError, STIXDeprecationWarning
 from ..properties import (
-    BooleanProperty, EnumProperty, FloatProperty, IDProperty, IntegerProperty,
-    ListProperty, ObservableProperty, PatternProperty, ReferenceProperty,
-    StringProperty, TimestampProperty, TypeProperty,
+    BinaryProperty, BooleanProperty, EmbeddedObjectProperty, EnumProperty,
+    FloatProperty, IDProperty, IntegerProperty, ListProperty,
+    ObservableProperty, PatternProperty, ReferenceProperty, StringProperty,
+    TimestampProperty, TypeProperty,
 )
 from ..utils import NOW
 from .common import ExternalReference, GranularMarking, KillChainPhase
@@ -26,19 +29,20 @@ class AttackPattern(STIXDomainObject):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
         ('description', StringProperty()),
+        ('aliases', ListProperty(StringProperty)),
         ('kill_chain_phases', ListProperty(KillChainPhase)),
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -53,8 +57,8 @@ class Campaign(STIXDomainObject):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
@@ -68,12 +72,12 @@ class Campaign(STIXDomainObject):
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
     def _check_object_constraints(self):
-        super(self.__class__, self)._check_object_constraints()
+        super(Campaign, self)._check_object_constraints()
 
         first_seen = self.get('first_seen')
         last_seen = self.get('last_seen')
@@ -93,21 +97,61 @@ class CourseOfAction(STIXDomainObject):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
         ('description', StringProperty()),
+        ('action_type', StringProperty()),
+        ('os_execution_envs', ListProperty(StringProperty)),
+        ('action_bin', BinaryProperty()),
+        ('action_reference', EmbeddedObjectProperty(ExternalReference)),
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
+    def _check_object_constraints(self):
+        super(CourseOfAction, self)._check_object_constraints()
+
+        self._check_mutually_exclusive_properties(
+            ["action_bin", "action_reference"],
+            at_least_one=False,
+        )
+
+
+class Grouping(STIXDomainObject):
+    # TODO: Add link
+    """For more detailed information on this object's properties, see
+    `the STIX 2.1 specification <link here>`__.
+    """
+
+    _type = 'grouping'
+    _properties = OrderedDict([
+        ('type', TypeProperty(_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
+        ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
+        ('revoked', BooleanProperty(default=lambda: False)),
+        ('labels', ListProperty(StringProperty)),
+        ('confidence', IntegerProperty()),
+        ('lang', StringProperty()),
+        ('external_references', ListProperty(ExternalReference)),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('name', StringProperty()),
+        ('description', StringProperty()),
+        ('context', StringProperty(required=True)),
+        ('object_refs', ListProperty(ReferenceProperty(invalid_types=[""], spec_version='2.1'), required=True)),
+    ])
+
 
 class Identity(STIXDomainObject):
     # TODO: Add link
@@ -119,8 +163,8 @@ class Identity(STIXDomainObject):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
@@ -134,7 +178,7 @@ class Identity(STIXDomainObject):
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -149,15 +193,17 @@ class Indicator(STIXDomainObject):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty()),
         ('description', StringProperty()),
         ('indicator_types', ListProperty(StringProperty, required=True)),
         ('pattern', PatternProperty(required=True)),
-        ('valid_from', TimestampProperty(default=lambda: NOW)),
+        ('pattern_type', StringProperty(required=True)),
+        ('pattern_version', StringProperty()),
+        ('valid_from', TimestampProperty(default=lambda: NOW, required=True)),
         ('valid_until', TimestampProperty()),
         ('kill_chain_phases', ListProperty(KillChainPhase)),
         ('revoked', BooleanProperty(default=lambda: False)),
@@ -165,12 +211,12 @@ class Indicator(STIXDomainObject):
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
     def _check_object_constraints(self):
-        super(self.__class__, self)._check_object_constraints()
+        super(Indicator, self)._check_object_constraints()
 
         valid_from = self.get('valid_from')
         valid_until = self.get('valid_until')
@@ -180,6 +226,47 @@ class Indicator(STIXDomainObject):
             raise ValueError(msg.format(self))
 
 
+class Infrastructure(STIXDomainObject):
+    # TODO: Add link
+    """For more detailed information on this object's properties, see
+    `the STIX 2.1 specification <link here>`__.
+    """
+
+    _type = 'infrastructure'
+    _properties = OrderedDict([
+        ('type', TypeProperty(_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
+        ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
+        ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
+        ('revoked', BooleanProperty(default=lambda: False)),
+        ('labels', ListProperty(StringProperty)),
+        ('confidence', IntegerProperty()),
+        ('lang', StringProperty()),
+        ('external_references', ListProperty(ExternalReference)),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('name', StringProperty(required=True)),
+        ('description', StringProperty()),
+        ('infrastructure_types', ListProperty(StringProperty, required=True)),
+        ('aliases', ListProperty(StringProperty)),
+        ('kill_chain_phases', ListProperty(KillChainPhase)),
+        ('first_seen', TimestampProperty()),
+        ('last_seen', TimestampProperty()),
+    ])
+
+    def _check_object_constraints(self):
+        super(Infrastructure, self)._check_object_constraints()
+
+        first_seen = self.get('first_seen')
+        last_seen = self.get('last_seen')
+
+        if first_seen and last_seen and last_seen < first_seen:
+            msg = "{0.id} 'last_seen' must be greater than or equal to 'first_seen'"
+            raise ValueError(msg.format(self))
+
+
 class IntrusionSet(STIXDomainObject):
     # TODO: Add link
     """For more detailed information on this object's properties, see
@@ -190,8 +277,8 @@ class IntrusionSet(STIXDomainObject):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
@@ -208,12 +295,12 @@ class IntrusionSet(STIXDomainObject):
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
     def _check_object_constraints(self):
-        super(self.__class__, self)._check_object_constraints()
+        super(IntrusionSet, self)._check_object_constraints()
 
         first_seen = self.get('first_seen')
         last_seen = self.get('last_seen')
@@ -233,10 +320,11 @@ class Location(STIXDomainObject):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
+        ('name', StringProperty()),
         ('description', StringProperty()),
         ('latitude', FloatProperty(min=-90.0, max=90.0)),
         ('longitude', FloatProperty(min=-180.0, max=180.0)),
@@ -252,12 +340,12 @@ class Location(STIXDomainObject):
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
     def _check_object_constraints(self):
-        super(self.__class__, self)._check_object_constraints()
+        super(Location, self)._check_object_constraints()
 
         if self.get('precision') is not None:
             self._check_properties_dependency(['longitude', 'latitude'], ['precision'])
@@ -265,6 +353,20 @@ class Location(STIXDomainObject):
         self._check_properties_dependency(['latitude'], ['longitude'])
         self._check_properties_dependency(['longitude'], ['latitude'])
 
+        if not (
+            'region' in self
+            or 'country' in self
+            or (
+                'latitude' in self
+                and 'longitude' in self
+            )
+        ):
+            raise PropertyPresenceError(
+                "Location objects must have the properties 'region', "
+                "'country', or 'latitude' and 'longitude'",
+                Location,
+            )
+
     def to_maps_url(self, map_engine="Google Maps"):
         """Return URL to this location in an online map engine.
 
@@ -326,23 +428,91 @@ class Malware(STIXDomainObject):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
-        ('name', StringProperty(required=True)),
+        ('name', StringProperty()),
         ('description', StringProperty()),
         ('malware_types', ListProperty(StringProperty, required=True)),
+        ('is_family', BooleanProperty(required=True)),
+        ('aliases', ListProperty(StringProperty)),
         ('kill_chain_phases', ListProperty(KillChainPhase)),
+        ('first_seen', TimestampProperty()),
+        ('last_seen', TimestampProperty()),
+        ('os_execution_envs', ListProperty(StringProperty)),
+        ('architecture_execution_envs', ListProperty(StringProperty)),
+        ('implementation_languages', ListProperty(StringProperty)),
+        ('capabilities', ListProperty(StringProperty)),
+        ('sample_refs', ListProperty(ReferenceProperty(valid_types=['artifact', 'file'], spec_version='2.1'))),
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
+    def _check_object_constraints(self):
+        super(Malware, self)._check_object_constraints()
+
+        first_seen = self.get('first_seen')
+        last_seen = self.get('last_seen')
+
+        if first_seen and last_seen and last_seen < first_seen:
+            msg = "{0.id} 'last_seen' must be greater than or equal to 'first_seen'"
+            raise ValueError(msg.format(self))
+
+        if self.is_family and "name" not in self:
+            raise PropertyPresenceError(
+                "'name' is a required property for malware families",
+                Malware,
+            )
+
+
+class MalwareAnalysis(STIXDomainObject):
+    # TODO: Add link
+    """For more detailed information on this object's properties, see
+    `the STIX 2.1 specification <link here>`__.
+    """
+
+    _type = 'malware-analysis'
+    _properties = OrderedDict([
+        ('type', TypeProperty(_type)),
+        ('spec_version', StringProperty(fixed='2.1')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
+        ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
+        ('revoked', BooleanProperty(default=lambda: False)),
+        ('labels', ListProperty(StringProperty)),
+        ('confidence', IntegerProperty()),
+        ('lang', StringProperty()),
+        ('external_references', ListProperty(ExternalReference)),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
+        ('granular_markings', ListProperty(GranularMarking)),
+        ('product', StringProperty(required=True)),
+        ('version', StringProperty()),
+        ('host_vm_ref', ReferenceProperty(valid_types='software', spec_version='2.1')),
+        ('operating_system_ref', ReferenceProperty(valid_types='software', spec_version='2.1')),
+        ('installed_software_refs', ListProperty(ReferenceProperty(valid_types='software', spec_version='2.1'))),
+        ('configuration_version', StringProperty()),
+        ('modules', ListProperty(StringProperty)),
+        ('analysis_engine_version', StringProperty()),
+        ('analysis_definition_version', StringProperty()),
+        ('submitted', TimestampProperty()),
+        ('analysis_started', TimestampProperty()),
+        ('analysis_ended', TimestampProperty()),
+        ('av_result', StringProperty()),
+        ('analysis_sco_refs', ListProperty(ReferenceProperty(valid_types="only_SCO", spec_version='2.1'))),
+    ])
+
+    def _check_object_constraints(self):
+        super(MalwareAnalysis, self)._check_object_constraints()
+
+        self._check_at_least_one_property(["av_result", "analysis_sco_refs"])
+
 
 class Note(STIXDomainObject):
     # TODO: Add link
@@ -354,20 +524,20 @@ class Note(STIXDomainObject):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('abstract', StringProperty()),
         ('content', StringProperty(required=True)),
         ('authors', ListProperty(StringProperty)),
-        ('object_refs', ListProperty(ReferenceProperty, required=True)),
+        ('object_refs', ListProperty(ReferenceProperty(invalid_types=[""], spec_version='2.1'), required=True)),
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -382,20 +552,21 @@ class ObservedData(STIXDomainObject):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('first_observed', TimestampProperty(required=True)),
         ('last_observed', TimestampProperty(required=True)),
         ('number_observed', IntegerProperty(min=1, max=999999999, required=True)),
-        ('objects', ObservableProperty(spec_version='2.1', required=True)),
+        ('objects', ObservableProperty(spec_version='2.1')),
+        ('object_refs', ListProperty(ReferenceProperty(valid_types="only_SCO_&_SRO", spec_version="2.1"))),
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -403,14 +574,17 @@ class ObservedData(STIXDomainObject):
         self.__allow_custom = kwargs.get('allow_custom', False)
         self._properties['objects'].allow_custom = kwargs.get('allow_custom', False)
 
+        if "objects" in kwargs:
+            warnings.warn(
+                "The 'objects' property of observed-data is deprecated in "
+                "STIX 2.1.",
+                STIXDeprecationWarning,
+            )
+
         super(ObservedData, self).__init__(*args, **kwargs)
 
     def _check_object_constraints(self):
-        super(self.__class__, self)._check_object_constraints()
-
-        if self.get('number_observed', 1) == 1:
-            self._check_properties_dependency(['first_observed'], ['last_observed'])
-            self._check_properties_dependency(['last_observed'], ['first_observed'])
+        super(ObservedData, self)._check_object_constraints()
 
         first_observed = self.get('first_observed')
         last_observed = self.get('last_observed')
@@ -419,6 +593,10 @@ class ObservedData(STIXDomainObject):
             msg = "{0.id} 'last_observed' must be greater than or equal to 'first_observed'"
             raise ValueError(msg.format(self))
 
+        self._check_mutually_exclusive_properties(
+            ["objects", "object_refs"],
+        )
+
 
 class Opinion(STIXDomainObject):
     # TODO: Add link
@@ -430,13 +608,12 @@ class Opinion(STIXDomainObject):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('explanation', StringProperty()),
         ('authors', ListProperty(StringProperty)),
-        ('object_refs', ListProperty(ReferenceProperty, required=True)),
         (
             'opinion', EnumProperty(
                 allowed=[
@@ -448,12 +625,13 @@ class Opinion(STIXDomainObject):
                 ], required=True,
             ),
         ),
+        ('object_refs', ListProperty(ReferenceProperty(invalid_types=[""], spec_version='2.1'), required=True)),
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -468,21 +646,21 @@ class Report(STIXDomainObject):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
         ('description', StringProperty()),
         ('report_types', ListProperty(StringProperty, required=True)),
         ('published', TimestampProperty(required=True)),
-        ('object_refs', ListProperty(ReferenceProperty, required=True)),
+        ('object_refs', ListProperty(ReferenceProperty(invalid_types=[""], spec_version='2.1'), required=True)),
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -502,14 +680,16 @@ class ThreatActor(STIXDomainObject):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
         ('description', StringProperty()),
         ('threat_actor_types', ListProperty(StringProperty, required=True)),
         ('aliases', ListProperty(StringProperty)),
+        ('first_seen', TimestampProperty()),
+        ('last_seen', TimestampProperty()),
         ('roles', ListProperty(StringProperty)),
         ('goals', ListProperty(StringProperty)),
         ('sophistication', StringProperty()),
@@ -522,10 +702,20 @@ class ThreatActor(STIXDomainObject):
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
+    def _check_object_constraints(self):
+        super(ThreatActor, self)._check_object_constraints()
+
+        first_observed = self.get('first_seen')
+        last_observed = self.get('last_seen')
+
+        if first_observed and last_observed and last_observed < first_observed:
+            msg = "{0.id} 'last_seen' must be greater than or equal to 'first_seen'"
+            raise ValueError(msg.format(self))
+
 
 class Tool(STIXDomainObject):
     # TODO: Add link
@@ -537,13 +727,14 @@ class Tool(STIXDomainObject):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
         ('description', StringProperty()),
         ('tool_types', ListProperty(StringProperty, required=True)),
+        ('aliases', ListProperty(StringProperty)),
         ('kill_chain_phases', ListProperty(KillChainPhase)),
         ('tool_version', StringProperty()),
         ('revoked', BooleanProperty(default=lambda: False)),
@@ -551,7 +742,7 @@ class Tool(STIXDomainObject):
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -566,8 +757,8 @@ class Vulnerability(STIXDomainObject):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('name', StringProperty(required=True)),
@@ -577,7 +768,7 @@ class Vulnerability(STIXDomainObject):
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -616,8 +807,8 @@ def CustomObject(type='x-custom-type', properties=None):
             [
                 ('type', TypeProperty(type)),
                 ('spec_version', StringProperty(fixed='2.1')),
-                ('id', IDProperty(type)),
-                ('created_by_ref', ReferenceProperty(type='identity')),
+                ('id', IDProperty(type, spec_version='2.1')),
+                ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
                 ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
                 ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
             ],
@@ -628,7 +819,7 @@ def CustomObject(type='x-custom-type', properties=None):
                 ('confidence', IntegerProperty()),
                 ('lang', StringProperty()),
                 ('external_references', ListProperty(ExternalReference)),
-                ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+                ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
                 ('granular_markings', ListProperty(GranularMarking)),
             ],
             sorted([x for x in properties if x[0].startswith('x_')], key=lambda x: x[0]),

stix2/v21/sro.py

@@ -17,18 +17,20 @@ class Relationship(STIXRelationshipObject):
     `the STIX 2.1 specification <link here>`__.
     """
 
+    _invalid_source_target_types = ['bundle', 'language-content', 'marking-definition', 'relationship', 'sighting']
+
     _type = 'relationship'
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('relationship_type', StringProperty(required=True)),
         ('description', StringProperty()),
-        ('source_ref', ReferenceProperty(required=True)),
-        ('target_ref', ReferenceProperty(required=True)),
+        ('source_ref', ReferenceProperty(invalid_types=_invalid_source_target_types, spec_version='2.1', required=True)),
+        ('target_ref', ReferenceProperty(invalid_types=_invalid_source_target_types, spec_version='2.1', required=True)),
         ('start_time', TimestampProperty()),
         ('stop_time', TimestampProperty()),
         ('revoked', BooleanProperty(default=lambda: False)),
@@ -36,7 +38,7 @@ class Relationship(STIXRelationshipObject):
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 
@@ -79,23 +81,24 @@ class Sighting(STIXRelationshipObject):
     _properties = OrderedDict([
         ('type', TypeProperty(_type)),
         ('spec_version', StringProperty(fixed='2.1')),
-        ('id', IDProperty(_type)),
-        ('created_by_ref', ReferenceProperty(type='identity')),
+        ('id', IDProperty(_type, spec_version='2.1')),
+        ('created_by_ref', ReferenceProperty(valid_types='identity', spec_version='2.1')),
         ('created', TimestampProperty(default=lambda: NOW, precision='millisecond')),
         ('modified', TimestampProperty(default=lambda: NOW, precision='millisecond')),
+        ('description', StringProperty()),
         ('first_seen', TimestampProperty()),
         ('last_seen', TimestampProperty()),
         ('count', IntegerProperty(min=0, max=999999999)),
-        ('sighting_of_ref', ReferenceProperty(required=True)),
-        ('observed_data_refs', ListProperty(ReferenceProperty(type='observed-data'))),
-        ('where_sighted_refs', ListProperty(ReferenceProperty(type='identity'))),
+        ('sighting_of_ref', ReferenceProperty(valid_types="only_SDO", spec_version='2.1', required=True)),
+        ('observed_data_refs', ListProperty(ReferenceProperty(valid_types='observed-data', spec_version='2.1'))),
+        ('where_sighted_refs', ListProperty(ReferenceProperty(valid_types='identity', spec_version='2.1'))),
         ('summary', BooleanProperty()),
         ('revoked', BooleanProperty(default=lambda: False)),
         ('labels', ListProperty(StringProperty)),
         ('confidence', IntegerProperty()),
         ('lang', StringProperty()),
         ('external_references', ListProperty(ExternalReference)),
-        ('object_marking_refs', ListProperty(ReferenceProperty(type='marking-definition'))),
+        ('object_marking_refs', ListProperty(ReferenceProperty(valid_types='marking-definition', spec_version='2.1'))),
         ('granular_markings', ListProperty(GranularMarking)),
     ])
 

stix2/version.py

@@ -1 +1 @@
-__version__ = "1.1.3"
+__version__ = "1.2.0"

stix2/workbench.py

@@ -20,6 +20,7 @@
 
 """
 
+import functools
 import stix2
 from . import AttackPattern as _AttackPattern
 from . import Campaign as _Campaign
@@ -52,6 +53,11 @@ from . import (  # noqa: F401
 )
 from .datastore.filters import FilterSet
 
+
+# Enable some adaptation to the current default supported STIX version.
+_STIX_VID = "v" + stix2.DEFAULT_VERSION.replace(".", "")
+
+
 # Use an implicit MemoryStore
 _environ = Environment(store=MemoryStore())
 
@@ -116,48 +122,39 @@ def _related_wrapper(self, *args, **kwargs):
     return _environ.related_to(self, *args, **kwargs)
 
 
-def _observed_data_init(self, *args, **kwargs):
-    self.__allow_custom = kwargs.get('allow_custom', False)
-    self._properties['objects'].allow_custom = kwargs.get('allow_custom', False)
-    super(self.__class__, self).__init__(*args, **kwargs)
-
-
-def _constructor_wrapper(obj_type):
-    # Use an intermediate wrapper class so the implicit environment will create objects that have our wrapper functions
-    class_dict = dict(
-        created_by=_created_by_wrapper,
-        relationships=_relationships_wrapper,
-        related=_related_wrapper,
-        **obj_type.__dict__
-    )
-
-    # Avoid TypeError about super() in ObservedData
-    if 'ObservedData' in obj_type.__name__:
-        class_dict['__init__'] = _observed_data_init
-
-    wrapped_type = type(obj_type.__name__, obj_type.__bases__, class_dict)
-
-    @staticmethod
-    def new_constructor(cls, *args, **kwargs):
-        x = _environ.create(wrapped_type, *args, **kwargs)
-        return x
-    return new_constructor
-
-
 def _setup_workbench():
-    # Create wrapper classes whose constructors call the implicit environment's create()
     for obj_type in STIX_OBJS:
-        new_class_dict = {
-            '__new__': _constructor_wrapper(obj_type),
-            '__doc__': 'Workbench wrapper around the `{0} <stix2.v20.sdo.rst#stix2.v20.sdo.{0}>`__ object. {1}'.format(obj_type.__name__, STIX_OBJ_DOCS),
-        }
-        new_class = type(obj_type.__name__, (), new_class_dict)
 
-        # Add our new class to this module's globals and to the library-wide mapping.
-        # This allows parse() to use the wrapped classes.
-        globals()[obj_type.__name__] = new_class
-        stix2.OBJ_MAP[obj_type._type] = new_class
-        new_class = None
+        # The idea here was originally to dynamically create subclasses which
+        # were cleverly customized such that instantiating them would actually
+        # invoke _environ.create().  This turns out to be impossible, since
+        # __new__ can never create the class in the normal way, since that
+        # invokes __new__ again, resulting in infinite recursion.  And
+        # _environ.create() does exactly that.
+        #
+        # So instead, we create something "class-like", in that calling it
+        # produces an instance of the desired class.  But these things will
+        # be functions instead of classes.  One might think this trickery will
+        # have undesirable side-effects, but actually it seems to work.
+        # So far...
+        new_class_dict = {
+            '__doc__': 'Workbench wrapper around the `{0} <stix2.{1}.sdo.rst#stix2.{1}.sdo.{0}>`__ object. {2}'.format(
+                obj_type.__name__,
+                _STIX_VID,
+                STIX_OBJ_DOCS,
+            ),
+            'created_by': _created_by_wrapper,
+            'relationships': _relationships_wrapper,
+            'related': _related_wrapper,
+        }
+
+        new_class = type(obj_type.__name__, (obj_type,), new_class_dict)
+        factory_func = functools.partial(_environ.create, new_class)
+
+        # Add our new "class" to this module's globals and to the library-wide
+        # mapping.  This allows parse() to use the wrapped classes.
+        globals()[obj_type.__name__] = factory_func
+        stix2.OBJ_MAP[obj_type._type] = factory_func
 
 
 _setup_workbench()

tox.ini

@@ -9,11 +9,11 @@ deps =
   pytest-cov
   coverage
   taxii2-client
+  pyjarowinkler
+  haversine
   medallion
 commands =
-  pytest --ignore=stix2/test/v20/test_workbench.py --ignore=stix2/test/v21/test_workbench.py --cov=stix2 stix2/test/ --cov-report term-missing
-  pytest stix2/test/v20/test_workbench.py --cov=stix2 --cov-report term-missing --cov-append
-  pytest stix2/test/v21/test_workbench.py --cov=stix2 --cov-report term-missing --cov-append
+  python -m pytest --cov=stix2 stix2/test/ --cov-report term-missing -W ignore::stix2.exceptions.STIXDeprecationWarning
 
 passenv = CI TRAVIS TRAVIS_*