- Added the methods
getMinArguments
andgetMaxArguments
to theFunction
class. These return the minimum and maximum positional arguments that the given function accepts.
MatchLiteralPattern
s such ascase None: ...
are now never pruned from the extracted source code. This fixes some situations where code was wrongly identified as unreachable.
No user-facing changes.
No user-facing changes.
- Fixed a bug in the extractor where a comment inside a subscript could sometimes cause the AST to be missing nodes.
- Using the
break
andcontinue
keywords outside of a loop, which is a syntax error but is accepted by our parser, would cause the control-flow construction to fail. This is now no longer the case.
- Deleted the old deprecated TypeTracking library.
- Deleted the deprecated
classRef
predicate from theFieldStorage
module, usesubclassRef
instead. - Deleted a lot of deprecated modules and predicates from
Stdlib.qll
, use API-graphs directly instead.
- Additional data flow models for the builtin functions
map
,filter
,zip
, andenumerate
have been added.
- The sensitive data library has been improved so that
snake_case
style variable names are recognized more reliably. This may result in more sensitive data being identified, and more results from queries that use the sensitive data library.
- Additional taint steps through methods of
lxml.etree.Element
andlxml.etree.ElementTree
objects from thelxml
PyPI package have been modeled.
- Added support for parameter annotations in API graphs. This means that in a function definition such as
def foo(x: Bar): ...
, you can now use thegetInstanceFromAnnotation()
method to step fromBar
tox
. In addition to this, thegetAnInstance
method now also includes instances arising from parameter annotations.
- Added modeling of
fastapi.Request
andstarlette.requests.Request
as sources of untrusted input, and modeling of tainted data flow out of these request objects.
- Deleted the old deprecated data flow API that was based on extending a configuration class. See https://github.blog/changelog/2023-08-14-new-dataflow-api-for-writing-custom-codeql-queries for instructions on migrating your queries to use the new API.
- Fixed a problem with the control-flow graph construction, where writing
case True:
orcase False:
would cause parts of the graph to be pruned by mistake.
- Added modeling of the
bottle
framework, leading to new remote flow sources and header writes
- Added partial support for the
copy.replace
method, added in Python 3.13. - Added support for type parameter defaults, as specified in PEP-696.
- Modelled that
re.finditer
returns an iterable ofre.Match
objects. This is now understood by the API graph in many cases. - Type tracking, and hence the API graph, is now able to correctly trace through comprehensions.
- More precise modelling of the dataflow through comprehensions. In particular, captured variables are now handled correctly.
- Dataflow out of yield is added, allowing proper tracing through generators.
- Added several models of standard library functions and classes, in anticipation of no longer extracting the standard library in a future release.
- Added support for custom threat-models, which can be used in most of our taint-tracking queries, see our documentation for more details.
- The common sanitizer guard
StringConstCompareBarrier
has been renamed toConstCompareBarrier
and expanded to cover comparisons with other constant values such asNone
. This may result in fewer false positive results for several queries.
- Deleted the deprecated
explorationLimit
predicate fromDataFlow::Configuration
, useFlowExploration<explorationLimit>
instead. - Deleted the deprecated
semmle.python.RegexTreeView
module, usesemmle.python.regexp.RegexTreeView
instead. - Deleted the deprecated
RegexString
class fromregex.qll
. - Deleted the deprecated
Regex
class, useRegExp
instead. - Deleted the deprecated
semmle/python/security/SQL.qll
file. - Deleted the deprecated
useSSL
predicates from the LDAP libraries, useuseSsl
instead.
No user-facing changes.
No user-facing changes.
- Added support for
DictionaryElement[<key>]
andDictionaryElementAny
when Customizing Library Models forsourceModel
(see https://codeql.github.com/docs/codeql-language-guides/customizing-library-models-for-python/)
- Additional modelling to detect direct writes to the
Set-Cookie
header has been added for several web frameworks.
- A number of Python queries now support sinks defined using data extensions. The format of data extensions for Python has been documented.
No user-facing changes.
No user-facing changes.
- CodeQL package management is now generally available, and all GitHub-produced CodeQL packages have had their version numbers increased to 1.0.0.
- A Python MaD (Models as Data) row may now contain a dotted path in the
type
column. Like in Ruby, a path to a class will refer to instances of that class. This means that the summary["foo", "Member[MyClass].Instance.Member[instance_method]", "Argument[0]", "ReturnValue", "value"]
can now be written["foo.MS_Class", "Member[instance_method]", "Argument[0]", "ReturnValue", "value"]
. To refer to an actual class, one may add a!
at the end of the path.
- The
request
parameter of FlaskSessionInterface.open_session
method is now modeled as a remote flow source. - Additional heuristics for a new sensitive data classification for private information (e.g. credit card numbers) have been added to the shared
SensitiveDataHeuristics.qll
library. This may result in additional results for queries that use sensitive data such aspy/clear-text-storage-sensitive-data
andpy/clear-text-logging-sensitive-data
.
- Added modeling of the
pyramid
framework, leading to new remote flow sources and sinks.
- Deleted the deprecated
RegExpPatterns
module fromRegexp.qll
. - Deleted the deprecated
Security/CWE-020/HostnameRegexpShared.qll
file.
- Renamed the
StrConst
class toStringLiteral
, for greater consistency with other languages. TheStrConst
andStr
classes are now deprecated and will be removed in a future release.
- Improved the type-tracking capabilities (and therefore also API graphs) to allow tracking items in tuples and dictionaries.
No user-facing changes.
No user-facing changes.
No user-facing changes.
- Fixed missing flow for dictionary updates (
d[<key>] = ...
) when<key>
is a string constant not used in dictionary literals or as name of keyword-argument. - Fixed flow for iterable unpacking (
a,b = my_tuple
) when it occurs on top-level (module) scope.
- The name "certification" is no longer seen as possibly being a certificate, and will therefore no longer be flagged in queries like "clear-text-logging" which look for sensitive data.
- Added modeling of the
psycopg
PyPI package as a SQL database library.
- Added
html.escape
as a sanitizer for HTML.
- Fixed the
a
(ASCII) inline flag not being recognized by the regular expression library.
- Deleted many deprecated predicates and classes with uppercase
LDAP
,HTTP
,URL
,CGI
etc. in their names. Use the PascalCased versions instead. - Deleted the deprecated
localSourceStoreStep
predicate, useflowsToStoreStep
instead. - Deleted the deprecated
iteration_defined_variable
predicate from theSSA
library. - Deleted various deprecated predicates from the points-to libraries.
- Deleted the deprecated
semmle/python/security/OverlyLargeRangeQuery.qll
,semmle/python/security/regexp/ExponentialBackTracking.qll
,semmle/python/security/regexp/NfaUtils.qll
, andsemmle/python/security/regexp/NfaUtils.qll
files. - The diagnostic query
py/diagnostics/successfully-extracted-files
, and therefore the Code Scanning UI measure of scanned Python files, now considers any Python file seen during extraction, even one with some errors, to be extracted / scanned.
- Added support for global data-flow through captured variables.
- Captured subclass relationships ahead-of-time for most popular PyPI packages so we are able to resolve subclass relationships even without having the packages installed. For example we have captured that
flask_restful.Resource
is a subclass offlask.views.MethodView
, so our Flask modeling will still consider a function namedpost
on aclass Foo(flask_restful.Resource):
as a HTTP request handler. - Python now makes use of the shared type tracking library, exposed as
semmle.python.dataflow.new.TypeTracking
. The existing type tracking library,semmle.python.dataflow.new.TypeTracker
, has consequently been deprecated.
- We would previously confuse all captured variables into a single scope entry node. Now they each get their own node so they can be tracked properly.
- The dataflow graph no longer contains SSA variables. Instead, flow is directed via the corresponding controlflow nodes. This should make the graph and the flow simpler to understand. Minor improvements in flow computation has been observed, but in general negligible changes to alerts are expected.
No user-facing changes.
- Added support for tarfile extraction filters as defined in PEP-706. In particular, calls to
TarFile.extract
, andTarFile.extractall
are no longer considered to be sinks for thepy/tarslip
query if a sufficiently safe filter is provided.
- Added modeling of
*args
and**kwargs
as routed-parameters in request handlers for django/flask/FastAPI/tornado.
- Added support for type parameters in function and class definitions, as well as the new Python 3.12 type alias statement.
- Added taint-flow modeling for regular expressions with
re
module from the standard library.
- Added basic flow for attributes defined on classes, when the attribute lookup is on a direct reference to that class (so not instance, cls parameter, or self parameter). Example: class definition
class Foo: my_tuples = (dangerous, safe)
and usageSINK(Foo.my_tuples[0])
.
- Added support for functions decorated with
contextlib.contextmanager
. - Namespace packages in the form of regular packages with missing
__init__.py
-files are now allowed. This enables the analysis to resolve modules and functions inside such packages.
- Added better support for API graphs when encountering
from ... import *
. For example in the codefrom foo import *; Bar()
, we will now find a result forAPI::moduleImport("foo").getMember("Bar").getACall()
- Deleted the deprecated
isBarrierGuard
predicate from the dataflow library and its uses, useisBarrier
and theBarrierGuard
module instead. - Deleted the deprecated
getAUse
,getAnImmediateUse
,getARhs
, andgetAValueReachingRhs
predicates from theAPI::Node
class. - Deleted the deprecated
fullyQualifiedToAPIGraphPath
class fromSubclassFinder.qll
, usefullyQualifiedToApiGraphPath
instead. - Deleted the deprecated
Paths.qll
file. - Deleted the deprecated
semmle.python.security.performance
folder, usesemmle.python.security.regexp
instead. - Deleted the deprecated
semmle.python.security.strings
andsemmle.python.web
folders. - Improved modeling of decoding through pickle related functions (which can lead to code execution), resulting in additional sinks for the Deserializing untrusted input query (
py/unsafe-deserialization
). Added support forpandas.read_pickle
,numpy.load
andjoblib.load
.
- Django Rest Framework better handles custom
ModelViewSet
classes functions - Regular expression fragments residing inside implicitly concatenated strings now have better location information.
- Subterms of regular expressions encoded as single-line string literals now have better source-location information.
No user-facing changes.
- Regular expressions containing multiple parse mode flags are now interpretted correctly. For example
"(?is)abc.*"
with both thei
ands
flags. - Added
shlex.quote
as a sanitizer for thepy/shell-command-constructed-from-input
query.
- Support analyzing packages (folders with python code) that do not have
__init__.py
files, although this is technically required, we see real world projects that don't have this. - Added modeling of AWS Lambda handlers that can be identified with
AWS::Serverless::Function
in YAML files, where the event parameter is modeled as a remote-flow-source. - Improvements of the
aiohttp
models including remote-flow-sources from type annotations, new path manipulation, and SSRF sinks.
- Fixed the computation of locations for imports with aliases in jump-to-definition.
No user-facing changes.
- The
DataFlow::StateConfigSig
signature module has gained default implementations forisBarrier/2
andisAdditionalFlowStep/4
. Hence it is no longer needed to providenone()
implementations of these predicates if they are not needed.
- Data flow configurations can now include a predicate
neverSkip(Node node)
in order to ensure inclusion of certain nodes in the path explanations. The predicate defaults to the end-points of the additional flow steps provided in the configuration, which means that such steps now always are visible by default in path explanations. - Add support for Models as Data for Reflected XSS query
- Parameters with a default value are now considered a
DefinitionNode
. This improvement was motivated by allowing type-tracking and API graphs to follow flow from such a default value to a use by a captured variable.
- It is now possible to specify flow summaries in the format "MyPkg;Member[list_map];Argument[1].ListElement;Argument[0].Parameter[0];value"
- Deleted many models that used the old dataflow library, the new models can be found in the
python/ql/lib/semmle/python/frameworks
folder. - More precise modeling of several container functions (such as
sorted
,reversed
) and methods (such asset.add
,list.append
). - Added modeling of taint flow through the template argument of
flask.render_template_string
andflask.stream_template_string
. - Deleted many deprecated predicates and classes with uppercase
API
,HTTP
,XSS
,SQL
, etc. in their names. Use the PascalCased versions instead. - Deleted the deprecated
getName()
predicate from theContainer
class, usegetAbsolutePath()
instead. - Deleted many deprecated module names that started with a lowercase letter, use the versions that start with an uppercase letter instead.
- Deleted many deprecated predicates in
PointsTo.qll
. - Deleted many deprecated files from the
semmle.python.security
package. - Deleted the deprecated
BottleRoutePointToExtension
class fromExtensions.qll
. - Type tracking is now aware of flow summaries. This leads to a richer API graph, and may lead to more results in some queries.
No user-facing changes.
No user-facing changes.
- Type tracking is now aware of reads of captured variables (variables defined in an outer scope). This leads to a richer API graph, and may lead to more results in some queries.
- Added more content-flow/field-flow for dictionaries, by adding support for reads through
mydict.get("key")
andmydict.setdefault("key", value)
, and store steps throughdict["key"] = value
andmydict.setdefault("key", value)
.
- Added support for querying the contents of YAML files.
- The recently introduced new data flow and taint tracking APIs have had a number of module and predicate renamings. The old APIs remain in place for now.
- Added modeling of SQL execution in the packages
sqlite3.dbapi2
,cassandra-driver
,aiosqlite
, and the functionssqlite3.Connection.executescript
/sqlite3.Cursor.executescript
andasyncpg.connection.connect()
. - Fixed module resolution so we allow imports of definitions that have had an attribute assigned to it, such as
class Foo; Foo.bar = 42
.
- Fixed some accidental predicate visibility in the backwards-compatible wrapper for data flow configurations. In particular,
DataFlow::hasFlowPath
,DataFlow::hasFlow
,DataFlow::hasFlowTo
, andDataFlow::hasFlowToExpr
were accidentally exposed in a single version.
No user-facing changes.
- Added support for merging two
PathGraph
s via disjoint union to allow results from multiple data flow computations in a singlepath-problem
query.
- The main data flow and taint tracking APIs have been changed. The old APIs remain in place for now and translate to the new through a backwards-compatible wrapper. If multiple configurations are in scope simultaneously, then this may affect results slightly. The new API is quite similar to the old, but makes use of a configuration module instead of a configuration class.
- Deleted the deprecated
getPath
andgetFolder
predicates from theXmlFile
class.
- We use a new analysis for the call-graph (determining which function is called). This can lead to changed results. In most cases this is much more accurate than the old call-graph that was based on points-to, but we do lose a few valid edges in the call-graph, especially around methods that are not defined inside its class.
- Fixed module resolution so we properly recognize definitions made within if-then-else statements.
- Added modeling of cryptographic operations in the
hmac
library.
- Python 2 is no longer supported for extracting databases using the CodeQL CLI. As a consequence,
the previously deprecated support for
pyxl
andspitfire
templates has also been removed. When extracting Python 2 code, having Python 2 installed is still recommended, as this ensures the correct version of the Python standard library is extracted.
- Fixed module resolution so we properly recognize that in
from <pkg> import *
, where<pkg>
is a package, the actual imports are made from the<pkg>/__init__.py
file.
No user-facing changes.
No user-facing changes.
- The PAM authorization bypass due to incorrect usage (
py/pam-auth-bypass
) query has been converted to a taint-tracking query, resulting in significantly fewer false positives.
- Added
subprocess.getoutput
andsubprocess.getoutputstatus
as new command injection sinks for the StdLib. - The data-flow library has been rewritten to no longer rely on the points-to analysis in order to resolve references to modules. Improvements in the module resolution can lead to more results.
- Deleted the deprecated
importNode
predicate from theDataFlowUtil.qll
file. - Deleted the deprecated features from
PEP249.qll
that were not inside thePEP249
module. - Deleted the deprecated
werkzeug
from theWerkzeug
module inWerkzeug.qll
. - Deleted the deprecated
methodResult
predicate fromPEP249::Cursor
.
except*
is now supported.- The result of
Try.getAHandler
andTry.getHandler(<index>)
is no longer of typeExceptStmt
, as handlers may also beExceptGroupStmt
s (After Python 3.11 introduced PEP 654). Instead, it is of the new typeExceptionHandler
of whichExceptStmt
andExceptGroupStmt
are subtypes. To support selecting only one type of handler,Try.getANormalHandler
andTry.getAGroupHandler
have been added. Existing uses ofTry.getAHandler
for which it is important to select only normal handlers, will need to be updated toTry.getANormalHandler
.
No user-facing changes.
No user-facing changes.
- The ReDoS libraries in
semmle.code.python.security.regexp
have been moved to a shared pack inside theshared/
folder, and the previous location has been deprecated.
No user-facing changes.
- Fixed labels in the API graph pertaining to definitions of subscripts. Previously, these were found by
getMember
rather thangetASubscript
. - Added edges for indices of subscripts to the API graph. Now a subscripted API node will have an edge to the API node for the index expression. So if
foo
is matched by API nodeA
, then"key"
infoo["key"]
will be matched by the API nodeA.getIndex()
. This can be used to track the origin of the index. - Added member predicate
getSubscriptAt(API::Node index)
toAPI::Node
. LikegetASubscript()
, this will return an API node that matches a subscript of the node, but here it will be restricted to subscripts where the index matches theindex
parameter. - Added convenience predicate
getSubscript("key")
to obtain a subscript at a specific index, when the index happens to be a statically known string.
- Added the ability to refer to subscript operations in the API graph. It is now possible to write
response().getMember("cookies").getASubscript()
to find code likeresp.cookies["key"]
(assumingresponse
returns an API node for response objects). - Added modeling of creating Flask responses with
flask.jsonify
.
- Some unused predicates in
SsaDefinitions.qll
,TObject.qll
,protocols.qll
, and thepointsto/
folder have been deprecated. - Some classes/modules with upper-case acronyms in their name have been renamed to follow our style-guide. The old name still exists as a deprecated alias.
- Changed
CallNode.getArgByName
such that it has results for keyword arguments given after a dictionary unpacking argument, as thebar=2
argument infunc(foo=1, **kwargs, bar=2)
. getStarArg
member-predicate onCall
andCallNode
has been changed for calls that have multiple*args
arguments (for examplefunc(42, *my_args, *other_args)
): Instead of producing no results, it will always have a result for the first such*args
argument.- Reads of global/non-local variables (without annotations) inside functions defined on classes now works properly in the case where the class had an attribute defined with the same name as the non-local variable.
- Fixed an issue in the taint tracking analysis where implicit reads were not allowed by default in sinks or additional taint steps that used flow states.
- Many classes/predicates/modules with upper-case acronyms in their name have been renamed to follow our style-guide. The old name still exists as a deprecated alias.
- The utility files previously in the
semmle.python.security.performance
package have been moved to thesemmle.python.security.regexp
package.
The previous files still exist as deprecated aliases.
- Most deprecated predicates/classes/modules that have been deprecated for over a year have been deleted.
- Change
.getASubclass()
onAPI::Node
so it allows to follow subclasses even if the class has a class decorator.
- The documentation of API graphs (the
API
module) has been expanded, and some of the members predicates ofAPI::Node
have been renamed as follows:getAnImmediateUse
->asSource
getARhs
->asSink
getAUse
->getAValueReachableFromSource
getAValueReachingRhs
->getAValueReachingSink
- Improved modeling of sensitive data sources, so common words like
certain
andsecretary
are no longer considered a certificate and a secret (respectively).
- The
BarrierGuard
class has been deprecated. Such barriers and sanitizers can now instead be created using the newBarrierGuard
parameterized module.
API::moduleImport
no longer has any results for dotted names, such asAPI::moduleImport("foo.bar")
. UsingAPI::moduleImport("foo.bar").getMember("baz").getACall()
previously worked if the Python code wasfrom foo.bar import baz; baz()
, but not if the code wasimport foo.bar; foo.bar.baz()
-- we are making this change to ensure the approach that can handle all cases is always used.
- The imports made available from
import python
are no longer exposed underDataFlow::
after doingimport semmle.python.dataflow.new.DataFlow
, for example usingDataFlow::Add
will now cause a compile error.
- The modeling of
request.files
in Flask has been fixed, so we now properly handle assignments to local variables (such asfiles = request.files; files['key'].filename
). - Added taint propagation for
io.StringIO
andio.BytesIO
. This addition was originally submitted as part of an experimental query by @jorgectf.
- The signature of
allowImplicitRead
onDataFlow::Configuration
andTaintTracking::Configuration
has changed fromallowImplicitRead(DataFlow::Node node, DataFlow::Content c)
toallowImplicitRead(DataFlow::Node node, DataFlow::ContentSet c)
.
- The recently added flow-state versions of
isBarrierIn
,isBarrierOut
,isSanitizerIn
, andisSanitizerOut
in the data flow and taint tracking libraries have been removed.
- Queries importing a data-flow configuration from
semmle.python.security.dataflow
should ensure that the imported file ends withQuery
, and only import its top-level module. For example, a query that usedCommandInjection::Configuration
fromsemmle.python.security.dataflow.CommandInjection
should from now useConfiguration
fromsemmle.python.security.dataflow.CommandInjectionQuery
instead.
- Added data-flow for Django ORM models that are saved in a database (no
models.ForeignKey
support).
- Improved modeling of Flask
Response
objects, so passing a response body with the keyword argumentresponse
is now recognized.
- The flow state variants of
isBarrier
andisAdditionalFlowStep
are no longer exposed in the taint tracking library. TheisSanitizer
andisAdditionalTaintStep
predicates should be used instead.
- Many classes/predicates/modules that had upper-case acronyms have been renamed to follow our style-guide. The old name still exists as a deprecated alias.
- Some modules that started with a lowercase letter have been renamed to follow our style-guide. The old name still exists as a deprecated alias.
- The data flow and taint tracking libraries have been extended with versions of
isBarrierIn
,isBarrierOut
, andisBarrierGuard
, respectivelyisSanitizerIn
,isSanitizerOut
, andisSanitizerGuard
, that support flow states.
- All deprecated predicates/classes/modules that have been deprecated for over a year have been deleted.
- Added new SSRF sinks for
httpx
,pycurl
,urllib
,urllib2
,urllib3
, andlibtaxii
. This improvement was submitted by @haby0. - The regular expression parser now groups sequences of normal characters. This reduces the number of instances of
RegExpNormalChar
. - Fixed taint propagation for attribute assignment. In the assignment
x.foo = tainted
we no longer treat the entire objectx
as tainted, just because the attributefoo
contains tainted data. This leads to slightly fewer false positives. - Improved analysis of attributes for data-flow and taint tracking queries, so
getattr
/setattr
are supported, and a write to an attribute properly stops flow for the old value in that attribute. - Added post-update nodes (
DataFlow::PostUpdateNode
) for arguments in calls that can't be resolved.
- The old points-to based modeling has been deprecated. Use the new type-tracking/API-graphs based modeling instead.
- Moved the files defining regex injection configuration and customization, instead of
import semmle.python.security.injection.RegexInjection
please useimport semmle.python.security.dataflow.RegexInjection
(the same forRegexInjectionCustomizations
). - The
codeql/python-upgrades
CodeQL pack has been removed. All upgrades scripts have been merged into thecodeql/python-all
CodeQL pack.
- Added modeling of many functions from the
os
module that uses file system paths, such asos.stat
,os.chdir
,os.mkdir
, and so on. - Added modeling of the
tempfile
module for creating temporary files and directories, such as the functionstempfile.NamedTemporaryFile
andtempfile.TemporaryDirectory
. - Extended the modeling of FastAPI such that custom subclasses of
fastapi.APIRouter
are recognized. - Extended the modeling of FastAPI such that
fastapi.responses.FileResponse
are consideredFileSystemAccess
. - Added modeling of the
posixpath
,ntpath
, andgenericpath
modules for path operations (although these are not supposed to be used), resulting in new sinks. - Added modeling of
wsgiref.simple_server
applications, leading to new remote flow sources.
- Added modeling of
os.stat
,os.lstat
,os.statvfs
,os.fstat
, andos.fstatvfs
, which are new sinks for the Uncontrolled data used in path expression (py/path-injection
) query. - Added modeling of the
posixpath
,ntpath
, andgenericpath
modules for path operations (although these are not supposed to be used), resulting in new sinks for the Uncontrolled data used in path expression (py/path-injection
) query. - Added modeling of
wsgiref.simple_server
applications, leading to new remote flow sources. - Added modeling of
aiopg
for sinks executing SQL. - Added modeling of HTTP requests and responses when using
flask_admin
(Flask-Admin
PyPI package), which leads to additional remote flow sources. - Added modeling of the PyPI package
toml
, which provides encoding/decoding of TOML documents, leading to new taint-tracking steps.