xdev package

Subpackages

Submodules

Module contents

This is Jon Crall’s xdev module.

These are tools I often use in IPython, but they almost never make it into production code, otherwise they would be in ubelt.

Read the docs

https://xdev.readthedocs.io

Github

https://github.com/Erotemic/xdev

Pypi

https://pypi.org/project/xdev
class xdev.AsciiDirectedGlyphs[source]

Bases: _AsciiBaseGlyphs

last = 'L-> '
mid = '|-> '
backedge = '<-'
vertical_edge = 'v'
class xdev.AsciiUndirectedGlyphs[source]

Bases: _AsciiBaseGlyphs

last = 'L-- '
mid = '|-- '
backedge = '-'
vertical_edge = '|'
class xdev.AvailablePackageConfig(*args: Any, **kwargs: Any)[source]

Bases: DataConfig

Print a table of available versions of a python package on Pypi

Refactor of ~/local/tools/supported_python_versions_pip.py to report the available versions of a python package that meet some critera

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

default = {'package_name': <Value(None)>, 'refresh': <Value(False)>, 'request_min': <Value(None)>}
class xdev.ChDir(dpath)[source]

Bases: object

Context manager that changes the current working directory and then returns you to where you were.

Parameters:

dpath (PathLike | None) – The new directory to work in. If None, then the context manager is disabled.

Example

>>> dpath = ub.Path.appdir('xdev/tests/chdir').ensuredir()
>>> dir1 = (dpath / 'dir1').ensuredir()
>>> dir2 = (dpath / 'dir2').ensuredir()
>>> with ChDir(dpath):
>>>     assert ub.Path.cwd() == dpath
>>>     # changes to the given directory, and then returns back
>>>     with ChDir(dir1):
>>>         assert ub.Path.cwd() == dir1
>>>         with ChDir(dir2):
>>>             assert ub.Path.cwd() == dir2
>>>             # changes inside the context manager will be reset
>>>             os.chdir(dpath)
>>>         assert ub.Path.cwd() == dir1
>>>     assert ub.Path.cwd() == dpath
>>>     with ChDir(dir1):
>>>         assert ub.Path.cwd() == dir1
>>>         with ChDir(None):
>>>             assert ub.Path.cwd() == dir1
>>>             # When disabled, the cwd does *not* reset at context exit
>>>             os.chdir(dir2)
>>>         assert ub.Path.cwd() == dir2
>>>         os.chdir(dir1)
>>>         # Dont change dirs, but reset to your cwd at context end
>>>         with ChDir('.'):
>>>             os.chdir(dir2)
>>>         assert ub.Path.cwd() == dir1
>>>     assert ub.Path.cwd() == dpath
class xdev.DirectoryStatsCLI(*args: Any, **kwargs: Any)[source]

Bases: DataConfig

Analysis for code in a repository

CommandLine

python ~/code/xdev/xdev/cli/repo_stats.py .

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

classmethod _register_main(func)[source]
default = {'dpath': <Value('.')>, 'exclude_dnames': <Value(None)>, 'exclude_fnames': <Value(None)>, 'ignore_dotprefix': <Value(True)>, 'include_dnames': <Value(None)>, 'include_fnames': <Value(None)>, 'max_display_depth': <Value(None)>, 'max_files': <Value(None)>, 'max_walk_depth': <Value(None)>, 'parse_content': <Value(False)>, 'python': <Value(False)>, 'rust': <Value(False)>, 'verbose': <Value(0)>, 'version': <Value(False)>}
main(**kwargs)

Example

>>> # xdoctest: +SKIP
>>> cmdline = 0
>>> kwargs = dict(dpath='module:watch')
>>> main(cmdline=cmdline, **kwargs)
normalize()
class xdev.DirectoryWalker(dpath, exclude_dnames=None, exclude_fnames=None, include_dnames=None, include_fnames=None, max_walk_depth=None, max_files=None, parse_content=False, show_progress=True, ignore_empty_dirs=False, sort=False, fs=None, **kwargs)[source]

Bases: object

Configurable directory walker that can explore a directory and report information about its contents in a concise manner.

Options will impact how long this process takes based on how much data / metadata we need to parse out of the filesystem.

Parameters:

  • dpath (str | PathLike) – the path to walk

  • exclude_dnames (Coercable[MultiPattern]) – blocks directory names matching this pattern

  • exclude_fnames (Coercable[MultiPattern]) – blocks file names matching this pattern

  • include_dnames (Coercable[MultiPattern]) – if specified, excludes directories that do NOT match this pattern.

  • include_fnames (Coercable[MultiPattern]) – if specified, excludes files that do NOT match this pattern.

  • max_files (None | int) – ignore all files in directories with more than this number.

  • max_walk_depth (None | int) – how far to recurse

  • parse_content (bool) – if True, include content analysis

  • sort (bool) – if True, sort files and directories before adding them to the graph.

  • fs (fsspec.spec.AbstractFileSystem) – experimental: an fsspec filesystem

  • **kwargs – passed to label options

property root

Alias for self.dpath

write_network_text(**kwargs)[source]
write_report(max_nodes=10, **nxtxt_kwargs)[source]
Parameters:

**nxtxt_kwargs –

pathstring or file or callable or None

Filename or file handle for data output. if a function, then it will be called for each generated line. if None, this will default to “sys.stdout.write”

with_labelsbool | str

If True will use the “label” attribute of a node to display if it exists otherwise it will use the node value itself. If given as a string, then that attribute name will be used instead of “label”. Defaults to True.

sourcesList

Specifies which nodes to start traversal from. Note: nodes that are not reachable from one of these sources may not be shown. If unspecified, the minimal set of nodes needed to reach all others will be used.

max_depthint | None

The maximum depth to traverse before stopping. Defaults to None.

ascii_onlyBoolean

If True only ASCII characters are used to construct the visualization

endstring

The line ending character

vertical_chainsBoolean

If True, chains of nodes will be drawn vertically when possible.

Example

>>> # xdoctest: +REQUIRES(module:pandas)
>>> import xdev
>>> walker = xdev.DirectoryWalker.demo()
>>> walker.write_report(max_nodes=0)
build()[source]

Build the internal graph structure with requested metadata

stats(typed=False, root=None)[source]

Return stats about the directories starting at the root. Requires walker has been built. If root unspecified uses walker root

_inplace_filter_dnames(dnames)[source]
_inplace_filter_fnames(fnames)[source]
_walk()[source]
property file_paths
property dir_paths
_accum_stats()[source]
_update_stats()[source]
_update_stats2()[source]
_parallel_process_files(func, desc=None, max_workers=8, mode='thread')[source]

Applies a function to every node.

classmethod _reduce_stats(stats)[source]

Combines stats over the a prefix

classmethod _humanize_stats(stats, node_type, reduce_prefix=False)[source]
_find_duplicate_files()[source]
_update_path_metadata()[source]
_update_labels()[source]

Update how each node will be displayed

_sort()[source]
classmethod demo()[source]

Create a persistent demo directory tree and return a built walker.

The directory is created under ub.Path.appdir('directory_walker/demo') and is re-initialized on each call to keep doctests deterministic.

Returns:

DirectoryWalker

Example

>>> import xdev
>>> walker = xdev.DirectoryWalker.demo()
>>> walker.dpath.exists()
True
find(pattern, data=False, root=None, filetype=None)[source]

Search for nodes whose name matches a MultiPattern, optionally filtering by type.

Parameters:

  • pattern – Coerced via MultiPattern.coerce(pattern) and tested with pattern.match(node.name).

  • data (bool) – if True, also yield the node data dict

  • root (pathlib.Path | None) – if specified, search descendants of this node

  • filetype (Iterable[str] | None) –

    Iterable of type chars from {‘f’, ‘d’, ‘l’}:

    • ‘f’ = regular file

    • ‘d’ = directory

    • ‘l’ = symlink

    Examples: ‘f’, ‘fd’, {‘l’}, [‘f’,’l’].

Yields:

pathlib.Path | Tuple[pathlib.Path, dict]

Example

>>> import xdev
>>> walker = DirectoryWalker.demo()
>>> sorted(p.name for p in walker.find('foo.txt'))
['foo.txt']
>>> sorted(p.name for p in walker.find('foo*', filetype='f'))
['foo.md', 'foo.txt']
>>> sorted(p.name for p in walker.find('adir', filetype='d'))
['adir']
>>> # Best-effort: only assert something meaningful if the symlink exists
>>> links = list(walker.find('alink.txt', filetype='l'))
>>> (len(links) == 1)
True
find_one(pattern, data=False, root=None, filetype=None)[source]

Find exactly one node matching a pattern (and optional type filter).

Parameters:

  • pattern – Coerced via kwutil.MultiPattern.coerce(pattern) (see find).

  • data (bool) – if True, also return the node data dict.

  • root (pathlib.Path | None) – if specified, search descendants of this node.

  • filetype (Iterable[str] | None) – iterable of {‘f’,’d’,’l’} (see find).

Returns:

pathlib.Path | Tuple[pathlib.Path, dict]

Raises:

KeyError – if zero or multiple matches are found.

Example

>>> walker = DirectoryWalker.demo()
>>> walker.find_one('foo.txt').name
'foo.txt'
class xdev.EmbedOnException(before_embed=None)[source]

Bases: object

Context manager which embeds in ipython if an exception is thrown

SeeAlso:

embed()

__call__(before_embed=None)[source]
class xdev.ExtendedStubGenerator[source]

Bases: object

_hack_for_info(info)[source]
visit_func_def(o: None, is_abstract: bool = False, is_overload: bool = False) → None[source]
process_decorator(o) → None[source]
visit_class_def(o) → None[source]
class xdev.GrepResult(fpath, pattern=None)[source]

Bases: NiceRepr

Manage and format results from grep

append(lx, line)[source]
format_text(color=True)[source]
class xdev.InteractiveIter(iterable=None, enabled=True, startx=0, default_action='next', custom_actions=[], wraparound=False, display_item=False, verbose=True)[source]

Bases: object

Choose next value interactively

iterable should be a list, not a generator. sorry

CommandLine

xdoctest -m xdev.interactive_iter InteractiveIter:0 --interact
xdoctest -m xdev.interactive_iter InteractiveIter:1 --interact

Example

>>> # xdoctest: +REQUIRES(--interact)
>>> from xdev.interactive_iter import *  # NOQA
>>> iterable = [1, 2, 3]
>>> enabled = True
>>> startx = 0
>>> default_action = 'next'
>>> custom_actions = []
>>> wraparound = False
>>> display_item = True
>>> verbose = True
>>> iiter = InteractiveIter(iterable, enabled, startx, default_action, custom_actions, wraparound, display_item, verbose)
>>> for _ in iiter:
>>>     pass

Example

>>> # xdoctest: +REQUIRES(--interact)
>>> # Interactive matplotlib stuff
>>> from xdev.interactive_iter import *  # NOQA
>>> import kwimage
>>> import kwplot
>>> plt = kwplot.autoplt(verbose=3, force='Qt5Agg')
>>> plt.ion()
>>> keys = list(kwimage.grab_test_image.keys())
>>> iterable = [kwimage.grab_test_image(key) for key in keys]
>>> iiter = InteractiveIter(iterable)
>>> for img in iiter:
>>>     kwplot.imshow(img)
>>>     InteractiveIter.draw()
Parameters:

  • iterable (None) – (default = None)

  • enabled (bool) – (default = True)

  • startx (int) – (default = 0)

  • default_action (str) – (default = ‘next’)

  • custom_actions (list) – list of 4-tuple (name, actions, help, func) (default = [])

  • wraparound (bool) – (default = False)

  • display_item (bool) – (default = True)

  • verbose (bool) – verbosity flag(default = True)

classmethod eventloop(custom_actions=[])[source]

For use outside of iteration wrapping. Makes an interactive event loop custom_actions should be specified in format [dispname, keys, desc, func]

handle_ans(ans_)[source]

preforms an actionm based on a user answer

prompt()[source]
wait_for_input()[source]
__call__(iterable=None)[source]
classmethod draw()[source]

in the common case where InteractiveIter is used to view matplotlib figures, you will have to draw the figure manually. This is a helper for that task.

class xdev.MultiPattern(patterns, predicate)[source]

Bases: PatternBase, NiceRepr

Example

>>> dpath = ub.Path.appdir('xdev/tests/multipattern_paths').ensuredir().delete().ensuredir()
>>> (dpath / 'file0.txt').touch()
>>> (dpath / 'data0.dat').touch()
>>> (dpath / 'other0.txt').touch()
>>> ((dpath / 'dir1').ensuredir() / 'file1.txt').touch()
>>> ((dpath / 'dir2').ensuredir() / 'file2.txt').touch()
>>> ((dpath / 'dir2').ensuredir() / 'file3.txt').touch()
>>> ((dpath / 'dir1').ensuredir() / 'data.dat').touch()
>>> ((dpath / 'dir2').ensuredir() / 'data.dat').touch()
>>> ((dpath / 'dir2').ensuredir() / 'data.dat').touch()
>>> pat = MultiPattern.coerce(['*.txt'], 'glob')
>>> print(list(pat.paths(cwd=dpath)))
>>> pat = MultiPattern.coerce(['*0*', '**/*.txt'], 'glob')
>>> print(list(pat.paths(cwd=dpath, recursive=1)))
>>> pat = MultiPattern.coerce(['*.txt', '**/*.txt', '**/*.dat'], 'glob')
>>> print(list(pat.paths(cwd=dpath)))
match(text)[source]
paths(cwd=None, recursive=False)[source]
_squeeze()[source]
classmethod coerce(data, hint='auto', predicate='any')[source]
Parameters:

  • data (str | List | Pattern | PathLike | MultiPattern)

  • hint (str) – can be ‘glob’, ‘regex’, ‘strict’ or ‘auto’. In ‘auto’ we will use ‘glob’ if the input is a string and ‘*’ is in the pattern, otherwise we will use strict. Pattern inputs keep their existing interpretation.

Returns:

MultiPattern

Example

>>> pat = MultiPattern.coerce('foo*', 'glob')
>>> pat2 = MultiPattern.coerce(pat, 'regex')
>>> pat3 = MultiPattern.coerce([pat, pat], 'regex')
>>> pat4 = MultiPattern.coerce([ub.Path('bar*'), pat], 'regex')
>>> print('pat = {}'.format(ub.urepr(pat, nl=1)))
>>> print('pat2 = {}'.format(ub.urepr(pat2, nl=1)))
>>> print('pat3 = {!r}'.format(pat3))
>>> print('pat4 = {!r}'.format(pat4))

>>> pat00 = MultiPattern.coerce('foo', 'glob')
>>> pat01 = MultiPattern.coerce('foo*', 'glob')
>>> pat02 = MultiPattern.coerce('foo*', 'regex')
>>> pat5 = MultiPattern.coerce(['foo', 'foo*', pat, pat00, pat01, pat02])
>>> print(f'pat5={pat5}')

Example

>>> # Test all acceptable input types
>>> import itertools as it
>>> str_pat = 'pattern*'
>>> scalar_inputs = {
>>>     'str': str_pat,
>>>     'path': ub.Path(str_pat),
>>>     'pat': Pattern.coerce(str_pat),
>>>     'mpat': MultiPattern.coerce(str_pat)
>>> }
>>> # Test scalar input types
>>> scalar_outputs = {}
>>> for k, v in scalar_inputs.items():
>>>     scalar_outputs[k] = MultiPattern.coerce(v)
>>> print('scalar_outputs = {}'.format(ub.urepr(scalar_outputs, nl=1)))
>>> #
>>> # Test iterable input types
>>> multi_outputs = []
>>> for v in it.combinations(scalar_inputs.values(), 2):
>>>     multi_outputs.append(MultiPattern.coerce(v))
>>> for v in it.combinations(scalar_inputs.values(), 3):
>>>     multi_outputs.append(MultiPattern.coerce(v))
>>> # Higher order nesting test
>>> higher_order_output = MultiPattern.coerce(multi_outputs)
>>> print('higher_order_output = {}'.format(ub.urepr(higher_order_output, nl=1)))
class xdev.Pattern(pattern, backend)[source]

Bases: PatternBase, NiceRepr

Provides a common API to several common pattern matching syntaxes.

A general patterns class, which can use a backend from BACKENDS

Parameters:

  • pattern (str | object) – The pattern text or a precompiled backend pattern object

  • backend (str) – Code indicating what backend the pattern text should be interpereted with. See BACKENDS for available choices.

Notes

# BACKENDS

The glob backend uses the fnmatch module [fnmatch_docs]. The regex backend uses the Python re module. The strict backend uses the “==” string equality testing. The parse backend uses the parse module.

References

Example

>>> # Test Regex backend
>>> repat = Pattern.coerce('foo.*', 'regex')
>>> assert repat.match('foobar')
>>> assert not repat.match('barfoo')
>>> match = repat.search('baz-biz-foobar')
>>> match = repat.match('baz-biz-foobar')
>>> # Test Glob backend
>>> globpat = Pattern.coerce('foo*', 'glob')
>>> assert globpat.match('foobar')
>>> assert not globpat.match('barfoo')
>>> globpat = Pattern.coerce('[foo|bar]', 'glob')
>>> globpat.match('foo')

Example

>>> # xdoctest: +REQUIRES(module:parse)
>>> # Test parse backend
>>> pattern1 = Pattern.coerce('A {adjective} pattern', 'parse')
>>> result1 = pattern1.match('A cool pattern')
>>> print(f'result1.named = {ub.urepr(result1.named, nl=1)}')
>>> pattern2 = pattern1.to_regex()
>>> result2 = pattern2.match('A cool pattern')
to_regex()[source]

Returns an equivalent pattern with the regular expression backend

Returns:

Pattern

Example

>>> globpat = Pattern.coerce('foo*', 'glob')
>>> strictpat = Pattern.coerce('foo*', 'strict')
>>> repat1 = strictpat.to_regex()
>>> repat2 = globpat.to_regex()
>>> print(f'repat1={repat1}')
>>> print(f'repat2={repat2}')
classmethod from_regex(data, flags=0, multiline=False, dotall=False, ignorecase=False)[source]

Create a Pattern object with a regex backend.

classmethod from_glob(data)[source]

Create a Pattern object with a glob backend.

classmethod coerce_backend(data, hint='auto')[source]

Example

>>> assert Pattern.coerce_backend('foo', hint='auto') == 'strict'
>>> assert Pattern.coerce_backend('foo*', hint='auto') == 'glob'
>>> assert Pattern.coerce_backend(re.compile('foo*'), hint='auto') == 'regex'
classmethod coerce(data, hint='auto')[source]

Attempt to automatically determine the input data as the appropriate pattern. If it cannot be determined, then fallback to the hint.

Parameters:

  • data (str | Pattern | PathLike)

  • hint (str) – can be ‘glob’, ‘regex’, ‘strict’ or ‘auto’. In ‘auto’ we will use ‘glob’ if the input is a string and ‘*’ is in the pattern, otherwise we will use strict. Pattern inputs keep their existing interpretation.

Example

>>> pat = Pattern.coerce('foo*', 'glob')
>>> pat2 = Pattern.coerce(pat, 'regex')
>>> print('pat = {}'.format(ub.urepr(pat, nl=1)))
>>> print('pat2 = {}'.format(ub.urepr(pat2, nl=1)))
match(text)[source]
search(text)[source]
sub(repl, text, count=-1)[source]
Parameters:

  • repl (str) – text to insert in place of pattern

  • text (str) – text to be searched and modified

  • count (int) – if non-negative, the maximum number of replacements that will be made.

paths(cwd=None, recursive=False)[source]

Find paths in the filesystem that match this pattern

Yields:

ub.Path

class xdev.PatternBase[source]

Bases: object

Abstract class that defines the Pattern api

match(text)[source]
search(text)[source]
sub(repl, text)[source]
class xdev.PythonRegexBuilder[source]

Bases: RegexBuilder

Contains helper methods to construct a regex

Example

>>> b = PythonRegexBuilder()
>>> pat_text = b.lookbehind('_') + r'v\d+' + b.optional(b.lookahead('_'))
>>> pat = re.compile(pat_text)
>>> print(pat.search('_v321_').group())
v321
>>> print(pat.search('_v321').group())
v321
>>> print(pat.search('fdsfds_v321_fdsfsd').group())
v321
>>> print(pat.search('fdsfds_v321fdsfsd').group())
v321
>>> print(pat.search('fdsfdsv321fdsfsd'))
None

Example

>>> # Test multiple negative lookbehind
>>> b = PythonRegexBuilder()
>>> suffix = 'foo'
>>> neg_prefix1 = b.lookbehind('abc', positive=0)
>>> neg_prefix2 = b.lookbehind('efg', positive=0)
>>> pat1 = re.compile(neg_prefix1 + suffix)
>>> pat2 = re.compile(neg_prefix2 + suffix)
>>> patB = re.compile(neg_prefix1 + neg_prefix2 + suffix)
>>> cases = ['abcfoo', 'efgfoo', 'hijfoo', 'foo']
>>> print([bool(pat1.search(c)) for c in cases])
>>> print([bool(pat2.search(c)) for c in cases])
>>> print([bool(patB.search(c)) for c in cases])
[False, True, True, True]
[True, False, True, True]
[False, False, True, True]

References

https://www.dataquest.io/blog/regex-cheatsheet/ https://docs.python.org/3/library/re.html#regular-expression-syntax

python_patterns = [{'alias': ['nongreedy_kleene_star'], 'docs': 'non-greedily matches zero or more of the pattern to the left', 'key': 'nongreedy_zero_or_more', 'pattern': '*?'}, {'docs': 'The boundary at the start or end of a word', 'key': 'boundary', 'pattern': '\\b'}, {'key': 'non-boundary', 'pattern': '\\B'}, {'key': 'left-expr', 'pattern': '\\A'}, {'docs': 'Matches only at the end of the string', 'key': 'right-expr', 'pattern': '\\Z'}]
previous(min=None, max=None, exact=None, greedy=True)[source]

Match the previous pattern some number of times.

Parameters:

  • min (int | None) – minimum number of matches

  • max (int | None) – maximum number of matches

  • exact (int | None) – Specify exact number of matches. Mutex with minimum and max.

  • greedy (bool) – if True match as many as possible, otherwise match as few as possible

Example

>>> from xdev.regex_builder import *  # NOQA
>>> b = PythonRegexBuilder()
>>> assert b.previous(exact=1) == '{1}'
>>> assert b.previous(min=1, max=3) == '{1,3}'
>>> assert b.previous(min=1, max=3, greedy=False) == '{1,3}?'
>>> assert b.previous(max=3) == '{,3}'
>>> assert b.previous(min=3) == '{3,}'
>>> assert b.previous() == '*'
>>> assert b.previous(greedy=False) == '*?'

Example

>>> from xdev.regex_builder import *  # NOQA
>>> b = PythonRegexBuilder()
>>> assert re.compile('a' + b.previous(exact=2) + '$').match('aa')
>>> assert not re.compile('a' + b.previous(exact=2) + '$').match('aaa')
>>> assert not re.compile('a' + b.previous(exact=2) + '$').match('a')
class xdev.PythonVersions[source]

Bases: object

Class that contains information about different Python versions

resolve_pyversion(pyver, maximize=False)[source]
xdev.RE_Pattern

alias of Pattern

class xdev.RegexBuilder[source]

Bases: object

Notes

The way to have multiple negative look aheads/behinds is to change them together SO12689046

References

Example

b = RegexBuilder.coerce(‘python’) import re pat = re.compile(‘[A-Z-]+’)

common_patterns = [{'docs': 'An alphanumeric word, i.e. [a-zA-Z0-9_] (also matches unicode characters in Python)', 'key': 'word', 'pattern': '\\w'}, {'docs': 'Anything not a word', 'key': 'non-word', 'pattern': '\\W'}, {'docs': 'Any space character including: " " "\\t", "\\n", "\\r"', 'key': 'space', 'pattern': '\\s'}, {'docs': 'Any non-space character', 'key': 'non-space', 'pattern': '\\S'}, {'docs': 'any number 0-9', 'key': 'digit', 'pattern': '\\d'}, {'docs': 'any non-digit', 'key': 'digit', 'pattern': '\\D'}, {'alias': ['kleene_star'], 'docs': 'zero or more of the pattern to the left', 'key': 'zero_or_more', 'pattern': '*'}]
lookahead(pat, positive=True, mode='positive')[source]

A lookahead pattern that can be positive or negative

looklook

lookbehind(pat, positive=True)[source]

A lookbehind pattern that can be positive or negative

named_field(pat, name=None)[source]
bref_field(name)[source]
escape(pat)[source]
optional(pat)[source]
group(pat)[source]
oneof(*paterns)[source]
classmethod coerce(backend='python')[source]
property identifier

A word, except it must start with a letter or underscore (not a number)

References

https://stackoverflow.com/questions/5474008/regular-expression-to-confirm-whether-a-string-is-a-valid-python-identifier

Example

>>> from xdev.regex_builder import *  # NOQA
>>> b = PythonRegexBuilder()
>>> assert re.match(b.identifier, 'hello')
>>> assert re.match(b.identifier, 'hello')
>>> assert re.match(b.identifier, '𝛣_ello')
>>> assert re.match(b.identifier, 'h_1e8llo')
>>> assert not re.match(b.identifier, '1hello')
property hex

A case-independent hex character

property word
property whitespace
property nongreedy
property integer

Pattern to match an integer.

Example

>>> from xdev.regex_builder import *  # NOQA
>>> b = PythonRegexBuilder()
>>> pat = re.compile('^' + b.integer + '$')
>>> assert pat.match('3')
>>> assert pat.match('+3')
>>> assert pat.match('-32')
property number

Can match a generic floating point number

References

https://www.regular-expressions.info/floatingpoint.html

Example

>>> from xdev.regex_builder import *  # NOQA
>>> b = PythonRegexBuilder()
>>> pat = re.compile('^' + b.number + '$')
>>> assert pat.match('3.4')
>>> assert pat.match('3.4e-1')
>>> assert pat.match('3.4')
>>> assert pat.match('3.4e+1')
>>> assert not pat.match('3.4a+1')

>>> b = PythonRegexBuilder()
>>> num_part = b.named_field(b.number, name='number')
>>> space_part = b.named_field(' *', name='spaces')
>>> unit_part = b.named_field('.*', name='unit')
>>> pat = re.compile('^' + num_part + space_part + unit_part + '$')
>>> pat.match('3.4').groupdict()
>>> pat.match('3.1415 foobars').groupdict()
>>> pat.match('3.1415foobars').groupdict()
>>> pat.match('+3.1415e9foobars').groupdict()
class xdev.ReqPythonVersionSpec(pattern)[source]

Bases: object

For python_version specs in requirements files

Example

>>> pattern = '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*'
>>> other = '3.7.2'
>>> reqspec = ReqPythonVersionSpec(pattern)
>>> reqspec.highest_explicit()
>>> reqspec.matches('2.6')
>>> reqspec.matches('2.7')

Example

>>> self = ReqPythonVersionSpec('~=3.2')
>>> self.highest_explicit()
highest_explicit()[source]
matches(other)[source]
class xdev.UtfDirectedGlyphs[source]

Bases: _UtfBaseGlyphs

last = '└─╼ '
mid = '├─╼ '
backedge = '╾'
vertical_edge = '╽'
class xdev.UtfUndirectedGlyphs[source]

Bases: _UtfBaseGlyphs

last = '└── '
mid = '├── '
backedge = '─'
vertical_edge = '│'
class xdev.VimRegexBuilder[source]

Bases: RegexBuilder

https://dev.to/iggredible/learning-vim-regex-26ep

vim_patterns = [{'alias': ['nongreedy_kleene_star'], 'docs': 'non-greedily matches zero or more of the pattern to the left', 'key': 'nongreedy_zero_or_more', 'pattern': '\\{-}'}]
previous(min=None, max=None, exact=None, greedy=True)[source]

Match the previous pattern some number of times.

Parameters:

  • min (int | None) – minimum number of matches

  • max (int | None) – maximum number of matches

  • exact (int | None) – Specify exact number of matches. Mutex with minimum and max.

  • greedy (bool) – if True match as many as possible, otherwise match as few as possible

Example

>>> from xdev.regex_builder import *  # NOQA
>>> b = VimRegexBuilder()
>>> assert b.previous(exact=1) == r'\{1}'
>>> assert b.previous(min=1, max=3) == r'\{1,3}'
>>> assert b.previous(min=1, max=3, greedy=False) == r'\{-1,3}'
>>> assert b.previous(max=3) == r'\{,3}'
>>> assert b.previous(min=3) == r'\{3,}'
>>> assert b.previous() == '*'
>>> assert b.previous(greedy=False) == r'\{-}'
class xdev.XdevCLI(description: str = '', sub_clis: List[Dict[str, Any]] | None = None, version: str | None = None)[source]

Bases: ModalCLI

The XDEV CLI

A collection of excellent developer tools for excellent developers.

class InfoCLI(*args: Any, **kwargs: Any)[source]

Bases: DataConfig

Info about xdev

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

classmethod main(cmdline=False, **kwargs)[source]
default = {}
class CodeblockCLI(*args: Any, **kwargs: Any)[source]

Bases: DataConfig

Remove indentation from text.

Useful for writing subscripts (e.g. python -c code) in shell files without having to resort to ugly indentation.

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

classmethod main(cmdline=False, **kwargs)[source]

Example

>>> from xdev.cli.main import *  # NOQA
>>> CodeblockCLI.main(cmdline=0, text='foobar')
default = {'text': <Value('')>}
class SedCLI(*args: Any, **kwargs: Any)[source]

Bases: DataConfig

Search and replace text in files

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

classmethod main(cmdline=False, **kwargs)[source]
default = {'dirblocklist': <Value(None)>, 'dpath': <Value(None)>, 'dry': <Value('ask')>, 'exclude': <Value(None)>, 'include': <Value(None)>, 'recursive': <Value(True)>, 'regexpr': <Value('')>, 'repl': <Value('')>, 'verbose': <Value(2)>}
class FindCLI(*args: Any, **kwargs: Any)[source]

Bases: DataConfig

Find matching files or paths in a directory.

This is similar to the GNU find program, but written in Python. Important differences are that this program is:

  • has pattern first argument and uses the cwd by default.

  • recursive by default

  • has explicit include / exclude options

Example

xdev find “*.py”

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

classmethod main(cmdline=False, **kwargs)[source]
default = {'dirblocklist': <Value(None)>, 'dpath': <Value(None)>, 'exclude': <Value(None)>, 'followlinks': <Value(False)>, 'include': <Value(None)>, 'pattern': <Value('')>, 'recursive': <Value(True)>, 'type': <Value('f')>}
class TreeCLI(*args: Any, **kwargs: Any)[source]

Bases: DataConfig

List a directory like a tree

See also

The, xdev

Example

xdev tree .

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

classmethod main(cmdline=False, **kwargs)[source]
default = {'colors': <Value(False)>, 'cwd': <Value('.')>, 'dirblocklist': <Value(None)>, 'ignore_dotprefix': <Value(True)>, 'max_depth': <Value(None)>, 'max_files': <Value(100)>}
class PintCLI(*args: Any, **kwargs: Any)[source]

Bases: DataConfig

Converts one type of unit to another via the pint library.

See also

The, for, that, we, it, Example, -------------, xdev, xdev, xdev, xdev, xdev

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

classmethod main(cmdline=False, **kwargs)[source]
default = {'input_expr': <Value(None)>, 'output_unit': <Value(None)>, 'precision': <Value(2)>}
class PyfileCLI(*args: Any, **kwargs: Any)[source]

Bases: DataConfig

Prints the path corresponding to a Python module.

This uses the ubelt.modname_to_modpath mechanism that does not require importing of your package.

Alternatives

An alternative with no dependencies is to use the one-liner:

python -c “import <modname>; print(<modname>.__file__)”

Example Usage

xdev pyfile xdev xdev pyfile numpy

# Use this feature in scripts for developement to avoid referencing # machine-specific paths. MODPATH=$(xdev pyfile ubelt) echo “MODPATH = $MODPATH”

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

classmethod main(cmdline=False, **kwargs)[source]
default = {'modname': <Value(None)>}
class PyVersionCLI(*args: Any, **kwargs: Any)

Bases: DataConfig

Detect and print the version of a Python module or package.

Note

Different backends may produce different results, especially for packages that are in development and were installed in development mode.

An alternative with no dependencies is to use the one-liner:

python -c “import <modname>; print(<modname>.__version__)”

xdev pyversion xdev xdev pyversion numpy

# Both the module name and the package name can be used. xdev pyversion cv2 xdev pyversion opencv-python-headless

# For more verbose information add the verbose flag xdev pyversion opencv-python-headless –verbose

xdev pyversion xdev –backend=import xdev pyversion xdev –backend=importlib

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

default = {'backend': <Value('auto')>, 'modname': <Value(None)>, 'verbose': <Value(False)>}
classmethod main(cmdline=False, **kwargs)
class EditfileCLI(*args: Any, **kwargs: Any)[source]

Bases: DataConfig

Opens a file in your visual editor determined by the VISUAL environment variable.

If VISUAL is unspecified it attempts to default to the first known existing editor.

Example Usage

xdev edit xdev xdev edit numpy

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

classmethod main(cmdline=False, **kwargs)[source]
default = {'target': <Value(None)>}
class FormatQuotesCLI(*args: Any, **kwargs: Any)[source]

Bases: DataConfig

Use single quotes for code and double quotes for docs.

This is useful for “fixing” quotations after running a code formater like black on a module.

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

classmethod main(cmdline=False, **kwargs)[source]
default = {'diff': <Value(True)>, 'path': <Value('')>, 'recursive': <Value(True)>, 'verbose': <Value(3)>, 'write': <Value(False)>}
class FreshPyenvCLI(*args: Any, **kwargs: Any)[source]

Bases: DataConfig

Create a fresh environment in a docker container to test a Python package.

SeeAlso

The generic freshpyenv.sh bash script also installed with this package.

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

classmethod main(cmdline=False, **kwargs)[source]
default = {'image': <Value('__default__')>}
class DocstrStubgenCLI(*args: Any, **kwargs: Any)[source]

Bases: DataConfig

Generate Typed Stubs from Docstrings (experimental)

Note

This is an experimental command and currently requires a specialized patch to mypy to work correctly.

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

classmethod main(cmdline=False, **kwargs)[source]
default = {'module': <Value(None)>}
class AvailablePackageCLI(*args: Any, **kwargs: Any)[source]

Bases: DataConfig

Print a table of available versions of a python package on Pypi

Refactor of ~/local/tools/supported_python_versions_pip.py to report the available versions of a python package that meet some critera

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

classmethod main(cmdline=False, **kwargs)[source]
default = {'package_name': <Value(None)>, 'refresh': <Value(False)>, 'request_min': <Value(None)>}
class DirectoryStatsCLI(*args: Any, **kwargs: Any)

Bases: DataConfig

Analysis for code in a repository

CommandLine

python ~/code/xdev/xdev/cli/repo_stats.py .

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

classmethod _register_main(func)
default = {'dpath': <Value('.')>, 'exclude_dnames': <Value(None)>, 'exclude_fnames': <Value(None)>, 'ignore_dotprefix': <Value(True)>, 'include_dnames': <Value(None)>, 'include_fnames': <Value(None)>, 'max_display_depth': <Value(None)>, 'max_files': <Value(None)>, 'max_walk_depth': <Value(None)>, 'parse_content': <Value(False)>, 'python': <Value(False)>, 'rust': <Value(False)>, 'verbose': <Value(0)>, 'version': <Value(False)>}
main(**kwargs)

Example

>>> # xdoctest: +SKIP
>>> cmdline = 0
>>> kwargs = dict(dpath='module:watch')
>>> main(cmdline=cmdline, **kwargs)
normalize()
class RegexCLI(*args: Any, **kwargs: Any)[source]

Bases: DataConfig

Query the regex builder for help on the command line. By default prints useful regex constructs I have a hard time remembering.

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

classmethod main(cmdline=False, **kwargs)[source]
default = {'backend': <Value('python')>}
class CLIFormatterCLI(*args: Any, **kwargs: Any)

Bases: DataConfig

The idea is that we can ingest a dictionary, argv list, or a command line string and convert between any of these formats.

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

default = {'input': <Value(None)>, 'input_type': <Value('auto')>, 'output_type': <Value('all')>}
classmethod main(cmdline=1, **kwargs)

Example

>>> # xdoctest: +SKIP
>>> from cli_formatter import *  # NOQA
>>> cmdline = 0
>>> kwargs = dict()
>>> cls = CLIFormatterCLI
>>> cls.main(cmdline=cmdline, **kwargs)
expand_import_star

alias of ExpandImportStarCLI

xdev.bubbletext(text, font='cybermedium')[source]

Uses pyfiglet to create bubble text.

Parameters:

font (str) – default=cybermedium, other fonts include: cybersmall and cyberlarge.

References

http://www.figlet.org/

Example

>>> bubble_text = bubbletext('TESTING BUBBLE TEXT', font='cybermedium')
>>> print(bubble_text)
xdev.build_package_table(package_name, refresh=False)[source]
xdev.byte_str(num, unit='auto', precision=2)[source]

Automatically chooses relevant unit (KB, MB, or GB) for displaying some number of bytes.

Parameters:

  • num (int) – number of bytes

  • unit (str) – which unit to use, can be auto, B, KB, MB, GB, TB, PB, EB, ZB, or YB.

  • precision (int) – number of decimals of precision

References

https://en.wikipedia.org/wiki/Orders_of_magnitude_(data)

Returns:

string representing the number of bytes with appropriate units

Return type:

str

Example

>>> num_list = [1, 100, 1024,  1048576, 1073741824, 1099511627776]
>>> result = ub.urepr(list(map(byte_str, num_list)), nl=0)
>>> print(result)
['0.00 KB', '0.10 KB', '1.00 KB', '1.00 MB', '1.00 GB', '1.00 TB']
xdev.common_module_aliases()[source]
xdev.common_module_names()[source]

fpath = ub.grabdata(’https://raw.githubusercontent.com/hugovk/top-pypi-packages/main/top-pypi-packages.json’, expires=86400) fpath = ub.Path(fpath) import json data = json.loads(fpath.read_text()) for item in data[‘rows’][0:300]:

pkg_name = item[‘project’] if ‘-’ not in pkg_name:

print(f’{pkg_name!r},’)

xdev.common_unreferenced()[source]
xdev.conj_phrase(list_, cond='or')[source]

Joins a list of words using English conjunction rules

Parameters:

  • list_ (list) – of strings

  • cond (str) – a conjunction (or, and, but)

Returns:

the joined cconjunction phrase

Return type:

str

References

http://en.wikipedia.org/wiki/Conjunction_(grammar)

Example

>>> list_ = ['a', 'b', 'c']
>>> result = conj_phrase(list_, 'or')
>>> print(result)
a, b, or c
Example1:
>>> list_ = ['a', 'b']
>>> result = conj_phrase(list_, 'and')
>>> print(result)
a and b
xdev.cp_sorter(v)[source]
xdev.delete_unpaired_pyi_files(modpath)[source]

Cleanup pyi files corresponding to renamed or removed py files.

xdev.demo()[source]
xdev.difftext(text1, text2, context_lines=0, ignore_whitespace=False, colored=False, style='ndiff', fromfile='', tofile='')[source]

Uses difflib to return a difference string between two similar texts

Parameters:

  • text1 (str) – old text

  • text2 (str) – new text

  • context_lines (int) – number of lines of unchanged context

  • ignore_whitespace (bool)

  • colored (bool) – if true highlight the diff

  • style (str) – can be ndiff or unified (git style)

  • fromfile (str) – unified diff “old” header label

  • tofile (str) – unified diff “new” header label

Returns:

formatted difference text message

Return type:

str

References

http://www.java2s.com/Code/Python/Utility/IntelligentdiffbetweentextfilesTimPeters.htm

Example

>>> # build test data
>>> text1 = 'one\ntwo\nthree'
>>> text2 = 'one\ntwo\nfive'
>>> # execute function
>>> result = difftext(text1, text2)
>>> # verify results
>>> print(result)
- three
+ five

Example

>>> # build test data
>>> from xdev.misc import *  # NOQA
>>> text1 = 'one\ntwo\nthree\n3.1\n3.14\n3.1415\npi\n3.4\n3.5\n4'
>>> text2 = 'one\ntwo\nfive\n3.1\n3.14\n3.1415\npi\n3.4\n4'
>>> # execute function
>>> context_lines = 1
>>> result = difftext(text1, text2, context_lines, colored=True)
>>> print(result)
>>> #
>>> result = difftext(text1, text2, context_lines, colored=True, style='unified')
>>> print(result)

Example

>>> # build test data for a git-apply-able unified patch
>>> from xdev.misc import *  # NOQA
>>> text1 = 'alpha\nbeta\ngamma\n'
>>> text2 = 'alpha\nbeta\nGAMMA\ndelta\n'
>>> patch = difftext(text1, text2, context_lines=3, style='unified', colored=True,
...                  fromfile='a/example.txt', tofile='b/example.txt')
>>> print(patch)
>>> lines = patch.splitlines()
>>> assert lines[0] == '--- a/example.txt'
>>> assert lines[1] == '+++ b/example.txt'
>>> assert lines[2].startswith('@@')
xdev.distext(obj)[source]

Like dis.dis, but returns the text

Parameters:

obj – a compiled object (e.g. a function, class, generator, etc…)

Returns:

text of the python byte code

Return type:

str

Example

>>> from xdev.introspect import *  # NOQA
>>> import ubelt as ub
>>> code = ub.codeblock(
    '''
    def foo(a, b):
        print('hello')
        if __debug__:
            if a is None or b is None:
                raise ValueError
        print('world')
        # This is not entirely optimized away
        if __debug__ and (a is None or b is None):
            raise ValueError
        return a + b
    ''')
>>> obj = compile(code, filename='<memory>', mode='exec', dont_inherit=True, optimize=0)
>>> print(' ** UNOPTIMIZED')
>>> text = distext(obj)
>>> print(text)
>>> print(' ** OPTIMIZED')
>>> obj = compile(code, filename='<memory>', mode='exec', dont_inherit=True, optimize=1)
>>> text = distext(obj)
>>> print(text)
xdev.edit_distance(string1, string2)[source]

Edit distance algorithm. String1 and string2 can be either strings or lists of strings

Parameters:

  • string1 (str | List[str])

  • string2 (str | List[str])

Requirements:

pip install python-Levenshtein

Returns:

float | List[float] | List[List[float]]

Example

>>> # xdoctest: +REQUIRES(module:Levenshtein)
>>> string1 = 'hello world'
>>> string2 = ['goodbye world', 'rofl', 'hello', 'world', 'lowo']
>>> edit_distance(['hello', 'one'], ['goodbye', 'two'])
>>> edit_distance('hello', ['goodbye', 'two'])
>>> edit_distance(['hello', 'one'], 'goodbye')
>>> edit_distance('hello', 'goodbye')
>>> distmat = edit_distance(string1, string2)
>>> result = ('distmat = %s' % (ub.repr2(distmat),))
>>> print(result)
>>> [7, 9, 6, 6, 7]
xdev.editfile(fpath, verbose=True)[source]

Opens a file or code corresponding to a live python object in your preferred visual editor. This function is mainly useful in an interactive IPython session.

The visual editor is determined by the VISUAL environment variable. If this is not specified it defaults to gvim.

Parameters:

  • fpath (PathLike | ModuleType | str) – a file path or python module / function. If the input is a string it will interpret it either as a Path or a module name. Ambiguity is resolved by choosing a path if the string resolves to an existing path, and then checking if the string corresponds to a module name.

  • verbose (int) – verbosity

Example

>>> # xdoctest: +SKIP
>>> # This test interacts with a GUI frontend, not sure how to test.
>>> import xdev
>>> ub.editfile(xdev.misc.__file__)
>>> ub.editfile(xdev)
>>> ub.editfile(xdev.editfile)
xdev.embed(parent_locals=None, parent_globals=None, exec_lines=None, remove_pyqt_hook=True, n=0, debug=False)[source]

Starts interactive session. Similar to keyboard command in matlab. Wrapper around IPython.embed.

Note

Contains helper logic to allow the developer to more easilly exit the program if embed is called in a loop. Specifically if you want to quit and not embed again, then set qqq=1 before exiting the embed session.

SeeAlso:

embed_on_exception()

xdev.embed_if_requested(n=0)[source]

Calls xdev.embed conditionally based on the environment.

Useful in cases where you want to leave the embed call around, but you dont want it to trigger in normal circumstances.

Specifically, embed is only called if the environment variable XDEV_EMBED exists or if –xdev-embed is in sys.argv.

xdev.ensure_rng(rng=None, api='python')[source]

Coerces input into a random number generator.

This function is useful for ensuring that your code uses a controlled internal random state that is independent of other modules.

If the input is None, then a global random state is returned.

If the input is a numeric value, then that is used as a seed to construct a random state.

If the input is a random number generator, then another random number generator with the same state is returned. Depending on the api, this random state is either return as-is, or used to construct an equivalent random state with the requested api.

Parameters:

  • rng (int | float | None | numpy.random.RandomState | random.Random) – if None, then defaults to the global rng. Otherwise this can be an integer or a RandomState class. Defaults to the global random.

  • api (str) – specify the type of random number generator to use. This can either be ‘numpy’ for a numpy.random.RandomState object or ‘python’ for a random.Random object. Defaults to numpy.

Returns:

rng - either a numpy or python random number generator, depending on the setting of api.

Return type:

(numpy.random.RandomState | random.Random)

Example

>>> # xdoctest: +REQUIRES(module:numpy)
>>> from xdev.util_random import *  # NOQA
>>> from xdev.util_random import ensure_rng
>>> rng = ensure_rng(None)
>>> ensure_rng(0, 'python').randint(0, 1000)
864
>>> # xdoctest: +REQUIRES(module:numpy)
>>> import numpy as np
>>> ensure_rng(np.random.RandomState(1)).randint(0, 1000)
427

Example

>>> from xdev.util_random import *  # NOQA
>>> from xdev.util_random import ensure_rng
>>> num = 4
>>> print('--- Python as PYTHON ---')
>>> py_rng = random.Random(0)
>>> pp_nums = [py_rng.random() for _ in range(num)]
>>> print(pp_nums)
>>> print('--- Numpy as PYTHON ---')
>>> # xdoctest: +REQUIRES(module:numpy)
>>> import numpy as np
>>> np_rng = ensure_rng(random.Random(0), api='numpy')
>>> np_nums = [np_rng.rand() for _ in range(num)]
>>> print(np_nums)
>>> print('--- Numpy as NUMPY---')
>>> np_rng = np.random.RandomState(seed=0)
>>> nn_nums = [np_rng.rand() for _ in range(num)]
>>> print(nn_nums)
>>> print('--- Python as NUMPY---')
>>> py_rng = ensure_rng(np.random.RandomState(seed=0), api='python')
>>> pn_nums = [py_rng.random() for _ in range(num)]
>>> print(pn_nums)
>>> assert np_nums == pp_nums
>>> assert pn_nums == nn_nums

Example

>>> # Test that random modules can be coerced
>>> # xdoctest: +REQUIRES(module:numpy)
>>> from xdev.util_random import *  # NOQA
>>> import random
>>> import numpy as np
>>> ensure_rng(random, api='python')
>>> ensure_rng(random, api='numpy')
>>> ensure_rng(np.random, api='python')
>>> ensure_rng(np.random, api='numpy')
xdev.find(pattern=None, dpath=None, include=None, exclude=None, dirblocklist=None, type=None, recursive=True, followlinks=False)[source]

Find all paths in a root subject to a search criterion

Parameters:

  • pattern (str | Pattern | None) – The glob pattern the path name must match to be returned

  • dpath (str | Pattern | None) – The root directory to search. Can also be a filepath, in which case, that is the only filepath considered. NOTE: in the future, this argument may change to path to indicate specifying a filepath is allowed. Defaults to cwd.

  • include (str | List[str] | MultiPattern | None) – Pattern or list of patterns. If specified, search only files whose base name matches this pattern. By default the pattern is GLOB. This only applies to the final name. Directories that do not match this name will still be traversed.

  • exclude (str | List[str] | MultiPattern | None) – Pattern or list of patterns. Skip any file with a name suffix that matches the pattern. By default the pattern is GLOB. This ONLY applies to the final name. Directories that match an exclude pattern will still be traversed. Use dirblocklist to specify patterns to exclude intermediate directories from traversal.

  • dirblocklist (str | List[str] | MultiPattern | None) – Any directory name matching this pattern will be removed from traversal.

  • type (str | List[str] | None) – A list of 1 character codes indicating what types of file can be returned. Currently we only allow either “f” for file or “d” for directory. Symbolic links are not currently distinguished. In the future we may support posix codes, see [1]_ for details.

  • recursive – search all subdirectories recursively

  • followlinks (bool, default=False) – if True will follow directory symlinks

References

_[1] https://linuxconfig.org/identifying-file-types-in-linux

Todo

mindepth

maxdepth

ignore_case

regex_match

Example

>>> from xdev.search_replace import *  # NOQA
>>> from xdev.search_replace import _create_test_filesystem
>>> dpath = _create_test_filesystem()['root']
>>> paths = list(find(pattern='*', dpath=dpath))
>>> assert len(paths) == 5
>>> paths = list(find(pattern='*', dpath=dpath, type='f'))
>>> assert len(paths) == 4
xdev.fix_embed_globals()[source]

HACK adds current locals() to globals(). Can be dangerous.

References

https://github.com/ipython/ipython/issues/62

Solves the following issue:

def foo():
    x = 5
    # You embed here
    import xdev
    xdev.embed()

    '''
    Now you try and run this line manually but you get a NameError

    result = [x + i for i in range(10)]

    No problem, just use. It changes all local variables to globals
    xdev.fix_embed_globals()

    '''
    result = [x + i for i in range(10)]

foo()
xdev.format_quotes_in_file(fpath, diff=True, write=False, verbose=3)[source]

Autoformat quotation marks in Python files

Parameters:

  • fpath (str) – The file to format

  • diff (bool) – if True write the diff between old and new to stdout

  • write (bool) – if True write the modifications to disk

  • verbose (int) – verbosity level

xdev.format_quotes_in_text(text, backend='parso')[source]

Reformat text according to formatting rules

Parameters:

text (str) – python source code

Returns:

modified text

Return type:

str

Example

>>> # xdoctest: +REQUIRES(module:parso)
>>> from xdev.format_quotes import *  # NOQA
>>> text = ub.codeblock(
...     f'''
...     def func1():
...         {TRIPLE_SINGLE_QUOTE}
...         Fix this.
...         {TRIPLE_SINGLE_QUOTE}
...
...     def func2():
...         {TRIPLE_DOUBLE_QUOTE}
...         Leave the doctests alone!
...         {TRIPLE_DOUBLE_QUOTE}
...
...     string1 = "fix these"
...     string2 = "don't fix these"
...     string3 = {TRIPLE_SINGLE_QUOTE}this is ok{TRIPLE_SINGLE_QUOTE}
...     string4 = {TRIPLE_DOUBLE_QUOTE}fix this{TRIPLE_DOUBLE_QUOTE}
...
...     def func3():
...         inside_string1 = "fix these"
...         inside_string2 = "don't fix these"
...         inside_string3 = {TRIPLE_SINGLE_QUOTE}this is ok{TRIPLE_SINGLE_QUOTE}
...         inside_string4 = {TRIPLE_DOUBLE_QUOTE}fix this{TRIPLE_DOUBLE_QUOTE}
...     ''')
>>> print(text)
>>> fixed = format_quotes_in_text(text)
>>> print(fixed)
>>> import xdev
>>> fixed = format_quotes_in_text(text, backend='parso')
>>> print('----')
>>> print(xdev.difftext(text, fixed, colored=True))
>>> # xdoctest: +REQUIRES(module:redbaron)
>>> print('----')
>>> fixed = format_quotes_in_text(text, backend='redbaron')
>>> print('----')
>>> print(xdev.difftext(text, fixed, colored=True))
>>> print('----')
xdev.generate_network_text(graph, with_labels=True, sources=None, max_depth=None, ascii_only=False, vertical_chains=False)[source]

Generate lines in the “network text” format

This works via a depth-first traversal of the graph and writing a line for each unique node encountered. Non-tree edges are written to the right of each node, and connection to a non-tree edge is indicated with an ellipsis. This representation works best when the input graph is a forest, but any graph can be represented.

This notation is original to networkx, although it is simple enough that it may be known in existing literature. See #5602 for details. The procedure is summarized as follows:

1. Given a set of source nodes (which can be specified, or automatically discovered via finding the (strongly) connected components and choosing one node with minimum degree from each), we traverse the graph in depth first order.

  1. Each reachable node will be printed exactly once on it’s own line.

  2. Edges are indicated in one of three ways:

    a. a parent “L-style” connection on the upper left. This corresponds to a traversal in the directed DFS tree.

    b. a backref “<-style” connection shown directly on the right. For directed graphs, these are drawn for any incoming edges to a node that is not a parent edge. For undirected graphs, these are drawn for only the non-parent edges that have already been represented (The edges that have not been represented will be handled in the recursive case).

    c. a child “L-style” connection on the lower right. Drawing of the children are handled recursively.

4. The children of each node (wrt the directed DFS tree) are drawn underneath and to the right of it. In the case that a child node has already been drawn the connection is replaced with an ellipsis (”…”) to indicate that there is one or more connections represented elsewhere.

5. If a maximum depth is specified, an edge to nodes past this maximum depth will be represented by an ellipsis.

Parameters:

  • graph (nx.DiGraph | nx.Graph) – Graph to represent

  • with_labels (bool | str) – If True will use the “label” attribute of a node to display if it exists otherwise it will use the node value itself. If given as a string, then that attribte name will be used instead of “label”. Defaults to True.

  • sources (List) – Specifies which nodes to start traversal from. Note: nodes that are not reachable from one of these sources may not be shown. If unspecified, the minimal set of nodes needed to reach all others will be used.

  • max_depth (int | None) – The maximum depth to traverse before stopping. Defaults to None.

  • ascii_only (Boolean) – If True only ASCII characters are used to construct the visualization

  • vertical_chains (Boolean) – If True, chains of nodes will be drawn vertically when possible.

Yields:

  • str (a line of generated text)

  • Example – >>> # xdoctest: +REQUIRES(module:networkx) >>> graph = nx.path_graph(10) >>> graph.add_node(‘A’) >>> graph.add_node(‘B’) >>> graph.add_node(‘C’) >>> graph.add_node(‘D’) >>> graph.add_edge(9, ‘A’) >>> graph.add_edge(9, ‘B’) >>> graph.add_edge(9, ‘C’) >>> graph.add_edge(‘C’, ‘D’) >>> graph.add_edge(‘C’, ‘E’) >>> graph.add_edge(‘C’, ‘F’) >>> write_network_text(graph) ╙── 0

    └── 1
    └── 2
    └── 3
    └── 4
    └── 5
    └── 6
    └── 7
    └── 8
    └── 9

    ├── A ├── B └── C

    ├── D ├── E └── F

    >>> write_network_text(graph, vertical_chains=True)
╙── 0
    │
    1
    │
    2
    │
    3
    │
    4
    │
    5
    │
    6
    │
    7
    │
    8
    │
    9
    ├── A
    ├── B
    └── C
        ├── D
        ├── E
        └── F
xdev.generate_typed_stubs(modpath)[source]

Attempt to use google-style docstrings, xdoctest, and mypy to generate typed stub files.

Does not overwrite anything by itself.

Parameters:

modpath (PathLike) – path to the module to generate types for

Returns:

A dictionary mapping the path of each file to write to the text to be written.

Return type:

Dict[PathLike, str]

Notes

FIXME: This currently requires my hacked version of mypy

CommandLine

xdoctest -m /home/joncrall/code/xdev/xdev/cli/docstr_stubgen.py generate_typed_stubs --hacked

Example

>>> # xdoctest: +REQUIRES(module:mypy)
>>> # xdoctest: +REQUIRES(--hacked)
>>> from xdev.cli.docstr_stubgen import *  # NOQA
>>> import xdev
>>> import ubelt as ub
>>> from xdev.cli import docstr_stubgen
>>> modpath = ub.Path(docstr_stubgen.__file__)
>>> generated = generate_typed_stubs(modpath)
>>> text = generated[ub.peek(generated.keys())]
>>> assert 'PathLike' in text
>>> assert 'Dict' in text
>>> print(text)
xdev.get_func_kwargs(func, max_depth=None)[source]

Dynamically parse the kwargs accepted by this function.

This function uses Python signatures where possible, but it also uses heuristics by inspecting the way any keywords dictionary is used.

Parameters:

  • func (callable) – function to introspect kwargs from

  • max_depth (int, default=None) – by default we recursively parse any kwargs passed to subfunctions.

Example

>>> from xinspect.dynamic_kwargs import get_func_kwargs
>>> parsed_kwargs = get_func_kwargs(get_func_kwargs)
>>> assert parsed_kwargs == {'max_depth': None}
xdev.get_stack_frame(N=0, strict=True)[source]
Parameters:

  • N (int) – N=0 means the frame you called this function in. N=1 is the parent frame.

  • strict (bool) – (default = True)

xdev.grab_pypi_items(package_name, refresh=False)[source]

Get all the information about a package from pypi

xdev.graph_str(graph, with_labels=True, sources=None, write=None, ascii_only=False)[source]

Creates a nice utf8 representation of a forest

This function has been superseded by nx.readwrite.text.generate_network_text(), which should be used instead.

Parameters:

  • graph (nx.DiGraph | nx.Graph) – Graph to represent (must be a tree, forest, or the empty graph)

  • with_labels (bool) – If True will use the “label” attribute of a node to display if it exists otherwise it will use the node value itself. Defaults to True.

  • sources (List) – Mainly relevant for undirected forests, specifies which nodes to list first. If unspecified the root nodes of each tree will be used for directed forests; for undirected forests this defaults to the nodes with the smallest degree.

  • write (callable) – Function to use to write to, if None new lines are appended to a list and returned. If set to the print function, lines will be written to stdout as they are generated. If specified, this function will return None. Defaults to None.

  • ascii_only (Boolean) – If True only ASCII characters are used to construct the visualization

Returns:

utf8 representation of the tree / forest

Return type:

str | None

Example

>>> # xdoctest: +REQUIRES(module:networkx)
>>> graph = nx.balanced_tree(r=2, h=3, create_using=nx.DiGraph)
>>> print(graph_str(graph))
╙── 0
    ├─╼ 1
    │   ├─╼ 3
    │   │   ├─╼ 7
    │   │   └─╼ 8
    │   └─╼ 4
    │       ├─╼ 9
    │       └─╼ 10
    └─╼ 2
        ├─╼ 5
        │   ├─╼ 11
        │   └─╼ 12
        └─╼ 6
            ├─╼ 13
            └─╼ 14

>>> # xdoctest: +REQUIRES(module:networkx)
>>> graph = nx.balanced_tree(r=1, h=2, create_using=nx.Graph)
>>> print(graph_str(graph))
╙── 0
    └── 1
        └── 2

>>> # xdoctest: +REQUIRES(module:networkx)
>>> print(graph_str(graph, ascii_only=True))
+-- 0
    L-- 1
        L-- 2
xdev.grep(regexpr, dpath=None, include=None, exclude=None, recursive=True, dirblocklist=None, verbose=1)[source]

Execute a grep on multiple files.

Parameters:

  • regexpr (str | Pattern) – pattern to find

  • dpath (str | None) – passed to find().

  • include (str | List[str] | MultiPattern | None) – passed to find().

  • exclude (str | List[str] | MultiPattern | None) – passed to find().

  • recursive (bool) – passed to find().

  • dirblocklist (str | List[str] | MultiPattern | None) – passed to find().

  • verbose (int) – verbosity level

Return type:

List[GrepResult]

Example

>>> from xdev.search_replace import *  # NOQA
>>> from xdev.search_replace import _create_test_filesystem
>>> dpath = _create_test_filesystem()['root']
>>> grep('a', dpath=dpath)
xdev.grepfile(fpath, regexpr, verbose=1)[source]

Exceute grep on a single file

Parameters:

  • fpath (str | PathLike) – file to search

  • regexpr (str | Pattern) – pattern to find

  • verbose (int) – verbosity level

Returns:

None | GrepResult

Example

>>> from xdev.search_replace import *  # NOQA
>>> from xdev.search_replace import _create_test_filesystem
>>> fpath = _create_test_filesystem()['contents'][1]
>>> grep_result = grepfile(fpath, r'\bb\b')
>>> print('grep_result = {}'.format(grep_result))
xdev.greptext(text, regexpr, fpath=None, verbose=1)[source]

Exceute grep on text

Parameters:

  • text (str) – text to search

  • regexpr (str | Pattern) – pattern to find

  • verbose (int) – verbosity level

Returns:

None | GrepResult

xdev.hacked_typing_info(type_name)[source]
xdev.import_module_from_pyx(fname, dpath=None, error='raise', autojit=True, verbose=1, recompile=False, annotate=False)[source]

Attempts to import a module corresponding to a pyx file.

If the corresponding compiled module is not found, this can attempt to JIT-cythonize the pyx file.

Parameters:

  • fname (str) – The basename of the cython pyx file

  • dpath (str) – The directory containing the cython pyx file

  • error (str) – Can be “raise” or “ignore”

  • autojit (bool) – If True, we will cythonize and compile the pyx file if possible.

  • verbose (int) – verbosity level (higher is more verbose)

  • recompile (bool) – if True force recompile

Returns:

ModuleType | None – Returns the compiled and imported module if possible, otherwise None

Return type:

module

xdev.iter_object_tree(obj)[source]
xdev.knapsack(items, maxweight, method='iterative')[source]

Solve the knapsack problem by finding the most valuable subsequence of items subject that weighs no more than maxweight.

Parameters:

  • items (tuple) – is a sequence of tuples (value, weight, id_), where value is a number and weight is a non-negative integer, and id_ is an item identifier.

  • maxweight (numbers.Real) – a non-negative number indicating the maximum weight of items we can take.

Returns:

(total_value, items_subset) - a pair whose first element is the

sum of values in the most valuable subsequence, and whose second element is the subsequence. Subset may be different depending on implementation (ie top-odwn recusrive vs bottom-up iterative)

Return type:

tuple

References

http://codereview.stackexchange.com/questions/20569/dynamic-programming-solution-to-knapsack-problem http://stackoverflow.com/questions/141779/solving-the-np-complete-problem-in-xkcd http://www.es.ele.tue.nl/education/5MC10/Solutions/knapsack.pdf

Example

>>> # ENABLE_DOCTEST
>>> import ubelt as ub
>>> items = [(4, 12, 0), (2, 1, 1), (6, 4, 2), (1, 1, 3), (2, 2, 4)]
>>> maxweight = 15
>>> total_value1, items_subset1 = knapsack(items, maxweight, method='iterative')
>>> print('items_subset1 = {}'.format(ub.repr2(items_subset1, nl=1)))
>>> print('total_value1 = {}'.format(ub.repr2(total_value1, nl=1)))

Example

>>> # ENABLE_DOCTEST
>>> # Solve https://xkcd.com/287/
>>> weights = [2.15, 2.75, 3.35, 3.55, 4.2, 5.8] * 2
>>> items = [(w, w, i) for i, w in enumerate(weights)]
>>> maxweight = 15.05
>>> total_value1, items_subset1 = knapsack(items, maxweight, method='iterative')
>>> total_weight = sum([t[1] for t in items_subset1])
>>> print('total_weight = %r' % (total_weight,))
>>> print('items_subset1 = %r' % (items_subset1,))
>>> #assert items_subset1 == items_subset, 'NOT EQ\n%r !=\n%r' % (items_subset1, items_subset)
Timeit:

# >>> import ubelt as ub # >>> setup = ub.codeblock( # >>> ‘’’ # import ubelt as ut # weights = [215, 275, 335, 355, 42, 58] * 40 # items = [(w, w, i) for i, w in enumerate(weights)] # maxweight = 2505 # #import numba # #knapsack_numba = numba.autojit(knapsack_iterative) # #knapsack_numba = numba.autojit(knapsack_iterative_numpy) # ‘’’) # >>> # Test load time # >>> stmt_list1 = ub.codeblock( # >>> ‘’’ # knapsack_iterative(items, maxweight) # knapsack_ilp(items, maxweight) # #knapsack_numba(items, maxweight) # #knapsack_iterative_numpy(items, maxweight) # ‘’’).split(’n’)

# ut.util_dev.timeit_compare(stmt_list1, setup, int(5))

from xdev.algo import * # NOQA def knapsack_inputs(n):

rng = np.random items = [

(rng.randint(0, 10), rng.randint(0, 10), idx) for idx in range(n)]

return {‘items’: items, ‘maxweight’: 100}

def input_wrapper(func):

import functools @functools.wraps(func) def _wrap(kwargs):

return func(**kwargs)

return _wrap

import perfplot perfplot.show(

setup=knapsack_inputs, kernels=list(map(input_wrapper, [knapsack_iterative, knapsack_ilp])), n_range=[2 ** k for k in range(15)], xlabel=”maxweight”, equality_check=None,

)

xdev.knapsack_greedy(items, maxweight)[source]

non-optimal greedy version of knapsack algorithm does not sort input. Sort the input by largest value first if desired.

Parameters:

  • `items` (tuple) – is a sequence of tuples (value, weight, id_), where value is a scalar and weight is a non-negative integer, and id_ is an item identifier.

  • `maxweight` (scalar) – is a non-negative integer.

Example

>>> # ENABLE_DOCTEST
>>> items = [(4, 12, 0), (2, 1, 1), (6, 4, 2), (1, 1, 3), (2, 2, 4)]
>>> maxweight = 15
>>> total_value, items_subset = knapsack_greedy(items, maxweight)
>>> result =  'total_value = %r\n' % (total_value,)
>>> result += 'items_subset = %r' % (items_subset,)
>>> print(result)
total_value = 7
items_subset = [(4, 12, 0), (2, 1, 1), (1, 1, 3)]
xdev.knapsack_ilp(items, maxweight, verbose=False)[source]

solves knapsack using an integer linear program

Example

>>> # DISABLE_DOCTEST
>>> import ubelt as ub
>>> # Solve https://xkcd.com/287/
>>> weights = [2.15, 2.75, 3.35, 3.55, 4.2, 5.8, 6.55]
>>> values  = [2.15, 2.75, 3.35, 3.55, 4.2, 5.8, 6.55]
>>> indices = ['mixed fruit', 'french fries', 'side salad',
>>>            'hot wings', 'mozzarella sticks', 'sampler plate',
>>>            'barbecue']
>>> items = [(v, w, i) for v, w, i in zip(values, weights, indices)]
>>> #items += [(3.95, 3.95, 'mystery plate')]
>>> maxweight = 15.05
>>> verbose = True
>>> total_value, items_subset = knapsack_ilp(items, maxweight, verbose)
>>> print('items_subset = %s' % (ub.repr2(items_subset, nl=1),))
xdev.knapsack_iterative(items, maxweight)[source]
xdev.knapsack_iterative_int(items, maxweight)[source]

Iterative knapsack method

Math:

maximize sum_{i in T} v_i subject to sum_{i in T} w_i leq W

Notes

dpmat is the dynamic programming memoization matrix. dpmat[i, w] is the total value of the items with weight at most W T is idx_subset, the set of indicies in the optimal solution

Example

>>> # ENABLE_DOCTEST
>>> weights = [1, 3, 3, 5, 2, 1] * 2
>>> items = [(w, w, i) for i, w in enumerate(weights)]
>>> maxweight = 10
>>> items = [(.8, 700, 0)]
>>> maxweight = 2000
>>> print('maxweight = %r' % (maxweight,))
>>> print('items = %r' % (items,))
>>> total_value, items_subset = knapsack_iterative_int(items, maxweight)
>>> total_weight = sum([t[1] for t in items_subset])
>>> print('total_weight = %r' % (total_weight,))
>>> print('items_subset = %r' % (items_subset,))
>>> result =  'total_value = %.2f' % (total_value,)
>>> print(result)
total_value = 0.80
xdev.knapsack_iterative_numpy(items, maxweight)[source]

Iterative knapsack method

maximize sum_{i in T} v_i subject to sum_{i in T} w_i leq W

Notes

dpmat is the dynamic programming memoization matrix. dpmat[i, w] is the total value of the items with weight at most W T is the set of indicies in the optimal solution

xdev.load_snapshot(fpath, parent_globals=None)[source]

Loads a snapshot of a local state from disk.

Parameters:

  • fpath (str | PathLike) – the path to the snapshot file to load

  • parent_globals (dict | None) – The state dictionary to update. Should be given as globals(). If unspecified, it is inferred via frame inspection.

xdev.make_warnings_print_tracebacks()[source]

Makes warnings show tracebacks when displayed.

Applies a monkeypatch to warnings.formatwarning that includes a traceback in the displayed warning message.

Note

Use the context manager :class:WarningsWithTracebacks instead, or WarningsWithTracebacks.apply() if global modification is needed.

Example

>>> # xdoctest: +SKIP
>>> make_warnings_print_tracebacks()
>>> warnings.warn('This is a test warning')
xdev.minimum_cross_python_versions(package_name, request_min=None, refresh=False)[source]

package_name = ‘scipy’ request_min = None

package_name = ‘opencv-python-headless’

package_name = ‘numpy’ request_min = ‘1.21.0’

xdev.modpath_coerce(modpath_coercable)[source]

if modpath_coercable is a name, statically converts it to a path

Parameters:

modpath_coercable (str | PathLike | ModuleType) – something we can extract a path to a module from.

Returns:

the coerced modpath

Return type:

str

Example

>>> # xdoctest: +SKIP
>>> from xdev.cli.docstr_stubgen import *  # NOQA
>>> import xdev
>>> modpath_coercable = xdev
>>> modpath = modpath_coerce(modpath_coercable)
>>> print(f'modpath={modpath}')
>>> assert modpath_coerce(modpath) == modpath
>>> assert modpath_coerce(xdev.__name__) == modpath
xdev.nested_type(obj, unions=False)[source]

Compute the :module:`typing` compatible annotation type.

Parameters:

  • obj (Any) – a typing template based on a specific object

  • unions (bool) – if True use unions, otherwise use Any

Returns:

type code (might change to return actual type)

Return type:

str

Example

>>> obj = {'a': [1, 2], 'b': [3, 4, 5]}
>>> print(nested_type(obj))
Dict[str, List[int]]

>>> import numpy as np
>>> obj = {'b': {'a': 1.0, 'b': 'foo', 'c': np.array([1, 2])}}
>>> print(nested_type(obj, unions=True))
Dict[str, Dict[str, float | ndarray | str]]
xdev.number_of_decimals(num)[source]
Parameters:

num (float)

References

stackoverflow.com/questions/6189956/finding-decimal-places

Example

>>> # ENABLE_DOCTEST
>>> num = 15.05
>>> result = number_of_decimals(num)
>>> print(result)
2
xdev.parse_file_stats(fpath, parse_content=True, fs=None)[source]

Get information about a file, including things like number of code lines / documentation lines, if that sort of information is available.

xdev.parse_platform_tag(platform_tag)[source]

Parse finer grained information out of the package platform tag.

Example

>>> cases = [
>>>     'manylinux1_x86_64',
>>>     'macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64',
>>>     'manylinux1_i686',
>>>     'win32',
>>>     'win_amd64',
>>>     'macosx_10_9_x86_64',
>>>     'macosx_10_9_intel',
>>>     'macosx_10_6_intel',
>>>     'manylinux2010_i686',
>>>     'manylinux2010_x86_64',
>>>     'manylinux2014_aarch64',
>>>     'manylinux_2_12_i686.manylinux2010_i686',
>>>     'manylinux_2_12_x86_64.manylinux2010_x86_64',
>>>     'manylinux_2_17_aarch64.manylinux2014_aarch64',
>>>     'manylinux_2_5_i686.manylinux1_i686',
>>>     'manylinux_2_5_x86_64.manylinux1_x86_64',
>>>     'macosx_10_9_universal2',
>>>     'macosx_11_0_arm64',
>>>     'manylinux_2_17_x86_64.manylinux2014_x86_64',
>>>     'macosx_10_14_x86_64',
>>>     'macosx_10_15_x86_64',
>>>     'macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64']
>>> for platform_tag in cases:
>>>     plat_info = parse_platform_tag(platform_tag)
>>>     print(f'platform_tag={platform_tag}')
>>>     print('plat_info = {}'.format(ub.repr2(plat_info, nl=1)))
xdev.parse_wheel_name(fname)[source]
xdev.postprocess_hacks(text, mod)[source]
xdev.profile_globals()[source]

Adds the profile decorator to all global functions

xdev.profile_now(func)[source]

Wrap a function to print profile information after it is called.

Parameters:

func (Callable) – function to profile

Returns:

the wrapped function

Return type:

Callable

Example

>>> # xdoctest: +SKIP
>>> from xdev.profiler import *  # NOQA
>>> def func_to_profile():
>>>     list(range(10))
>>>     tuple(range(100))
>>>     set(range(1000))
>>> profile_now(func_to_profile)()  # xdoctest: +IGNORE_WANT

Timer unit: 1e-09 s

Total time: 2.7767e-05 s File: <ipython-input-11-049a3440df03> Function: func_to_profile at line 3

Line # Hits Time Per Hit % Time Line Contents

3 def func_to_profile(): 4 1 3200.0 3200.0 11.5 list(range(10)) 5 1 1949.0 1949.0 7.0 tuple(range(100)) 6 1 22618.0 22618.0 81.5 set(range(1000))

xdev.quantum_random(pure=False)[source]

Returns a quantum random number as a 32 bit unsigned integer. Does this by making a network request to the ANU Quantum Random Number Generator web service, so an internet connection is required.

Parameters:

pure (bool) – if False, mixes this data with pseudorandom data for security. Otherwise returns the raw quantum numbers that were sent over the web (i.e. subject to MitM attacks).

Requirements:

quantumrandom >= 1.9.0

Returns:

the random number

Return type:

numpy.uint32

xdev.reload_class(self, verbose=True, reload_module=True)[source]

Reload a class for a specific instance.

(populates any new methods that may have just been written)

Ported from utool

xdev.remove_duplicate_imports(text)[source]
xdev.rprint(*args)[source]
xdev.sed(regexpr, repl, dpath=None, include=None, exclude=None, dirblocklist=None, recursive=True, dry=False, verbose=1)[source]

Execute a sed on multiple files.

Parameters:

  • regexpr (str | Pattern) – pattern to find

  • repl (str) – the text to replace the found pattern with

  • dpath (str | None) – passed to find().

  • include (str | List[str] | MultiPattern | None) – passed to find().

  • exclude (str | List[str] | MultiPattern | None) – passed to find().

  • dirblocklist (str | List[str] | MultiPattern | None) – passed to find().

  • recursive (bool) – passed to find().

  • dry (bool) – if True does not apply edits

  • verbose (int) – verbosity level

Example

>>> from xdev.search_replace import *  # NOQA
>>> from xdev.search_replace import _create_test_filesystem
>>> dpath = _create_test_filesystem()['root']
>>> sed('a', 'x', dpath=dpath, dry=True)
xdev.sedfile(fpath, regexpr, repl, dry=False, verbose=1)[source]

Execute a search and replace on a particular file

Parameters:

  • fpath (str | PathLike) – file to search / replace on

  • regexpr (str | Pattern) – pattern to find

  • repl (str) – the text to replace the found pattern with

  • dry (bool) – if True does not apply edits

  • verbose (int) – verbosity level

Returns:

changed lines

Return type:

List[Tuple[str, str]]

Todo

  • [ ] Store “SedResult” class, with lazy execution

Example

>>> from xdev.search_replace import *  # NOQA
>>> from xdev.search_replace import _create_test_filesystem
>>> fpath = _create_test_filesystem()['contents'][1]
>>> changed_lines1 = sedfile(fpath, 'a', 'x', dry=True, verbose=1)
>>> changed_lines2 = sedfile(fpath, 'a', 'x', dry=False, verbose=0)
>>> assert changed_lines2 == changed_lines1
>>> changed_lines3 = sedfile(fpath, 'a', 'x', dry=False, verbose=0)
>>> assert changed_lines3 != changed_lines2
xdev.set_overlaps(set1, set2, s1='s1', s2='s2', n_samples=None)[source]

Return sizes about set overlaps.

If the inputs are not sets, they will be cast to sets, and if there are duplicate they will be counted.

Parameters:

  • set1 (Iterable) – the first set of items

  • set2 (Iterable) – the second set of items

  • s1 (str) – name for set1

  • s2 (str) – name for set2

  • n_samples (int | None) – provide up to n examples from each set.

Returns:

sizes of sets intersections unions and differences

Return type:

Dict[str, int] | Dict[str, int | Dict]

Example

>>> import ubelt as ub
>>> from xdev.misc import set_overlaps
>>> set1 = {'a', 'b', 'c', 'd', 'e'}
>>> set2 = {'a', 'e', 'i', 'o', 'u'}
>>> result = set_overlaps(set1, set2, 'first5', 'vowels')
>>> print(f'result = {ub.urepr(result, nl=1)}')
result = {
    'first5': 5,
    'vowels': 5,
    'isect': 2,
    'union': 8,
    'first5 - vowels': 3,
    'vowels - first5': 3,
}

Example

>>> import ubelt as ub
>>> from xdev.misc import set_overlaps
>>> set1 = [1, 1, 2, 3, 3, 3, 4, 5,]
>>> set2 = [2, 2, 2, 3]
>>> result = set_overlaps(set1, set2)
>>> print(f'result = {ub.urepr(result, nl=1)}')
xdev.sidecar_glob(main_pat, sidecar_ext, main_key='main', sidecar_key=None, recursive=0)[source]

Similar to a regular glob, but returns a dictionary with associated main-file / sidecar-file pairs.

Todo

add as a general option to Pattern.paths?

Parameters:

main_pat (str | PathLike) – glob pattern for the main non-sidecar file

Yields:

Dict[str, ub.Path | None]

Notes

A sidecar file is defined by the sidecar extension. We usually use this for .dvc sidecars.

When the pattern includes a .dvc suffix, the result will include those .dvc files and any matching main files they correspond to. Note: if you search for paths like foo_*.dvc this might skiped unstaged files. Therefore it is recommended to only include the .dvc suffix in the pattern ONLY if you do not want any unstaged files.

If you want both staged and unstaged files, ensure the pattern does not exclude objects without a .dvc suffix (i.e. don’t end the pattern with .dvc).

When the pattern does not include a .dvc suffix, we include all those files, for other files that exist by adding a .dvc suffix.

With the pattern matches both a dvc and non-dvc file, they are grouped together.

Example

>>> from xdev.util_path import *  # NOQA
>>> dpath = ub.Path.appdir('xdev/tests/sidecar_glob')
>>> dpath.delete().ensuredir()
>>> (dpath / 'file1').touch()
>>> (dpath / 'file1.ext').touch()
>>> (dpath / 'file1.ext.car').touch()
>>> (dpath / 'file2.ext').touch()
>>> (dpath / 'file3.ext.car').touch()
>>> (dpath / 'file4.car').touch()
>>> (dpath / 'file5').touch()
>>> (dpath / 'file6').touch()
>>> (dpath / 'file6.car').touch()
>>> (dpath / 'file7.bike').touch()
>>> def _handle_results(results):
...     results = list(results)
...     for row in results:
...         for k, v in row.items():
...             if v is not None:
...                 row[k] = v.relative_to(dpath)
...     print(ub.repr2(results, sv=1))
...     return results
>>> main_key = 'main',
>>> sidecar_key = '.car'
>>> sidecar_ext = '.car'
>>> main_pat = dpath / '*'
>>> _handle_results(sidecar_glob(main_pat, sidecar_ext))
>>> _handle_results(sidecar_glob(dpath / '*.ext', '.car'))
>>> _handle_results(sidecar_glob(dpath / '*.car', '.car'))
>>> _handle_results(sidecar_glob(dpath / 'file*.ext', '.car'))
>>> _handle_results(sidecar_glob(dpath / '*', '.ext'))
xdev.snapshot(parent_ns=None, n=0)[source]

Save a snapshot of the local state to disk.

Serialize all names in scope to a pickle and save to disk. Also print the command that will let the user start an IPython session with this namespace.

Parameters:

  • parent_ns (dict) – A dictionary containing all of the available names in the scope to export.

  • n (int) – if parent_ns is unspecified, infer locals and globals from the frame n stack levels above the namespace this function is called in.

Todo

  • [ ] need to handle __main__

References

xdev.startfile(fpath, verbose=True)[source]

Uses default program defined by the system to open a file. This is done via os.startfile on windows, open on mac, and xdg-open on linux.

Parameters:

  • fpath (PathLike) – a file to open using the program associated with the files extension type.

  • verbose (int) – verbosity

References

http://stackoverflow.com/questions/2692873/quote-posix

Example

>>> # xdoctest: +SKIP
>>> # This test interacts with a GUI frontend, not sure how to test.
>>> import ubelt as ub
>>> base = ub.ensure_app_cache_dir('ubelt')
>>> fpath1 = join(base, 'test_open.txt')
>>> ub.touch(fpath1)
>>> proc = ub.startfile(fpath1)
xdev.stdlib_names()[source]
xdev.strip_comments_and_newlines(source)[source]

Removes hashtag comments from underlying source

Parameters:

source (str | List[str])

Todo

would be better if this was some sort of configurable minify API

Example

>>> from xdev.directory_walker import strip_comments_and_newlines
>>> import ubelt as ub
>>> fmtkw = dict(sss=chr(39) * 3, ddd=chr(34) * 3)
>>> source = ub.codeblock(
>>>    '''
       # comment 1
       a = '# not a comment'  # comment 2

multiline_string = {ddd}

one

{ddd} b = [

1, # foo

# bar 3,

] c = 3 ‘’’).format(**fmtkw)

>>> non_comments = strip_comments_and_newlines(source)
>>> print(non_comments)
>>> assert non_comments.count(chr(10)) == 10
>>> assert non_comments.count('#') == 1
xdev.strip_docstrings(tokens)[source]

Replace docstring tokens with NL tokens in a tokenize stream.

Any STRING token not part of an expression is deemed a docstring. Indented docstrings are not yet recognised.

xdev.summarize_package_availability(package_name)[source]

Todo

for each released version of the package we want to know

  • For source distros:

    • Does it need to be compiled?

    • What are is the min (or max?) python version

  • For binaries:

    • What python version, arch, and os targets are available.

xdev.take_column(list_, colx)[source]

iterator version of take_column

xdev.test_object_pickleability(obj)[source]
xdev.textfind(text, pattern)[source]

Return a colored text that highlights the pattern

xdev.tree(path)[source]

Like os.walk but yields a flat list of file and directory paths

Parameters:

path (str | os.PathLike) – path to traverse

Yields:

str – path

Example

>>> import itertools as it
>>> import ubelt as ub
>>> path = ub.Path('.')
>>> gen = tree(path)
>>> results = list(it.islice(gen, 5))
>>> print('results = {}'.format(ub.repr2(results, nl=1)))
xdev.tree_repr(cwd=None, max_files=100, dirblocklist=None, show_nfiles='auto', return_text=False, return_tree=False, pathstyle='name', max_depth=None, with_type=False, abs_root_label=True, ignore_dotprefix=True, colors=False)[source]

Filesystem tree representation

Like the unix util tree, but allow writing numbers of files per directory when given -d option

Parameters:

  • cwd (None | str | PathLike) – directory to print

  • max_files (int | None) – maximum files to print before supressing a directory

  • pathstyle (str) – can be rel, name, or abs

  • return_tree (bool) – if True return the tree

  • return_text (bool) – if True return the text

  • maxdepth (int | None) – maximum depth to descend

  • abs_root_label (bool) – if True force the root to always be absolute

  • colors (bool) – if True use rich

SeeAlso:

xdev.tree - generator

xdev.vectorize(func)[source]
xdev.view_directory(dpath=None, verbose=False)[source]

View a directory in the operating system file browser. Currently supports windows explorer, mac open, and linux nautlius.

Parameters:

  • dpath (PathLike | None) – directory name

  • verbose (bool) – verbosity

xdev.write_network_text(graph, path=None, with_labels=True, sources=None, max_depth=None, ascii_only=False, end='\n', vertical_chains=False)[source]

Creates a nice text representation of a graph

This works via a depth-first traversal of the graph and writing a line for each unique node encountered. Non-tree edges are written to the right of each node, and connection to a non-tree edge is indicated with an ellipsis. This representation works best when the input graph is a forest, but any graph can be represented.

Parameters:

  • graph (nx.DiGraph | nx.Graph) – Graph to represent

  • path (string or file or callable or None) – Filename or file handle for data output. if a function, then it will be called for each generated line. if None, this will default to “sys.stdout.write”

  • with_labels (bool | str) – If True will use the “label” attribute of a node to display if it exists otherwise it will use the node value itself. If given as a string, then that attribte name will be used instead of “label”. Defaults to True.

  • sources (List) – Specifies which nodes to start traversal from. Note: nodes that are not reachable from one of these sources may not be shown. If unspecified, the minimal set of nodes needed to reach all others will be used.

  • max_depth (int | None) – The maximum depth to traverse before stopping. Defaults to None.

  • ascii_only (Boolean) – If True only ASCII characters are used to construct the visualization

  • end (string) – The line ending characater

  • vertical_chains (Boolean) – If True, chains of nodes will be drawn vertically when possible.

Example

>>> # xdoctest: +REQUIRES(module:networkx)
>>> graph = nx.balanced_tree(r=2, h=2, create_using=nx.DiGraph)
>>> write_network_text(graph)
╙── 0
    ├─╼ 1
    │   ├─╼ 3
    │   └─╼ 4
    └─╼ 2
        ├─╼ 5
        └─╼ 6

>>> # A near tree with one non-tree edge
>>> graph.add_edge(5, 1)
>>> write_network_text(graph)
╙── 0
    ├─╼ 1 ╾ 5
    │   ├─╼ 3
    │   └─╼ 4
    └─╼ 2
        ├─╼ 5
        │   └─╼  ...
        └─╼ 6

>>> graph = nx.cycle_graph(5)
>>> write_network_text(graph)
╙── 0
    ├── 1
    │   └── 2
    │       └── 3
    │           └── 4 ─ 0
    └──  ...

>>> graph = nx.cycle_graph(5, nx.DiGraph)
>>> write_network_text(graph, vertical_chains=True)
╙── 0 ╾ 4
    ╽
    1
    ╽
    2
    ╽
    3
    ╽
    4
    └─╼  ...

>>> write_network_text(graph, vertical_chains=True, ascii_only=True)
+-- 0 <- 4
    v
    1
    v
    2
    v
    3
    v
    4
    L->  ...

>>> graph = nx.generators.barbell_graph(4, 2)
>>> write_network_text(graph, vertical_chains=False)
╙── 4
    ├── 5
    │   └── 6
    │       ├── 7
    │       │   ├── 8 ─ 6
    │       │   │   └── 9 ─ 6, 7
    │       │   └──  ...
    │       └──  ...
    └── 3
        ├── 0
        │   ├── 1 ─ 3
        │   │   └── 2 ─ 0, 3
        │   └──  ...
        └──  ...
>>> write_network_text(graph, vertical_chains=True)
╙── 4
    ├── 5
    │   │
    │   6
    │   ├── 7
    │   │   ├── 8 ─ 6
    │   │   │   │
    │   │   │   9 ─ 6, 7
    │   │   └──  ...
    │   └──  ...
    └── 3
        ├── 0
        │   ├── 1 ─ 3
        │   │   │
        │   │   2 ─ 0, 3
        │   └──  ...
        └──  ...

>>> graph = nx.complete_graph(5, create_using=nx.Graph)
>>> write_network_text(graph)
╙── 0
    ├── 1
    │   ├── 2 ─ 0
    │   │   ├── 3 ─ 0, 1
    │   │   │   └── 4 ─ 0, 1, 2
    │   │   └──  ...
    │   └──  ...
    └──  ...

>>> graph = nx.complete_graph(3, create_using=nx.DiGraph)
>>> write_network_text(graph)
╙── 0 ╾ 1, 2
    ├─╼ 1 ╾ 2
    │   ├─╼ 2 ╾ 0
    │   │   └─╼  ...
    │   └─╼  ...
    └─╼  ...