xdev package¶

Subpackages¶

xdev.cli package
- Submodules
- Module contents

Submodules¶

Module contents¶

This is Jon Crall’s xdev module.

These are tools I often use in IPython, but they almost never make it into production code, otherwise they would be in ubelt.

Read the docs	https://xdev.readthedocs.io
Github	https://github.com/Erotemic/xdev
Pypi	https://pypi.org/project/xdev

class xdev.AsciiDirectedGlyphs[source]¶

Bases: _AsciiBaseGlyphs

last = 'L-> '¶

mid = '|-> '¶

backedge = '<-'¶

vertical_edge = 'v'¶

class xdev.AsciiUndirectedGlyphs[source]¶

Bases: _AsciiBaseGlyphs

last = 'L-- '¶

mid = '|-- '¶

backedge = '-'¶

vertical_edge = '|'¶

class xdev.ChDir(dpath)[source]¶

Bases: object

Context manager that changes the current working directory and then returns you to where you were.

Parameters:: dpath (PathLike | None) – The new directory to work in. If None, then the context manager is disabled.

Example

>>> dpath = ub.Path.appdir('xdev/tests/chdir').ensuredir()
>>> dir1 = (dpath / 'dir1').ensuredir()
>>> dir2 = (dpath / 'dir2').ensuredir()
>>> with ChDir(dpath):
>>>     assert ub.Path.cwd() == dpath
>>>     # changes to the given directory, and then returns back
>>>     with ChDir(dir1):
>>>         assert ub.Path.cwd() == dir1
>>>         with ChDir(dir2):
>>>             assert ub.Path.cwd() == dir2
>>>             # changes inside the context manager will be reset
>>>             os.chdir(dpath)
>>>         assert ub.Path.cwd() == dir1
>>>     assert ub.Path.cwd() == dpath
>>>     with ChDir(dir1):
>>>         assert ub.Path.cwd() == dir1
>>>         with ChDir(None):
>>>             assert ub.Path.cwd() == dir1
>>>             # When disabled, the cwd does *not* reset at context exit
>>>             os.chdir(dir2)
>>>         assert ub.Path.cwd() == dir2
>>>         os.chdir(dir1)
>>>         # Dont change dirs, but reset to your cwd at context end
>>>         with ChDir('.'):
>>>             os.chdir(dir2)
>>>         assert ub.Path.cwd() == dir1
>>>     assert ub.Path.cwd() == dpath

class xdev.EmbedOnException(before_embed=None)[source]¶

Bases: object

Context manager which embeds in ipython if an exception is thrown

SeeAlso:: embed()

class xdev.GrepResult(fpath, pattern=None)[source]¶

Bases: NiceRepr

Manage and format results from grep

append(lx, line)[source]¶

format_text(color=True)[source]¶

class xdev.InteractiveIter(iterable=None, enabled=True, startx=0, default_action='next', custom_actions=[], wraparound=False, display_item=False, verbose=True)[source]¶

Bases: object

Choose next value interactively

iterable should be a list, not a generator. sorry

CommandLine

xdoctest -m xdev.interactive_iter InteractiveIter:0 --interact
xdoctest -m xdev.interactive_iter InteractiveIter:1 --interact

Example

>>> # xdoctest: +REQUIRES(--interact)
>>> from xdev.interactive_iter import *  # NOQA
>>> iterable = [1, 2, 3]
>>> enabled = True
>>> startx = 0
>>> default_action = 'next'
>>> custom_actions = []
>>> wraparound = False
>>> display_item = True
>>> verbose = True
>>> iiter = InteractiveIter(iterable, enabled, startx, default_action, custom_actions, wraparound, display_item, verbose)
>>> for _ in iiter:
>>>     pass

Example

>>> # xdoctest: +REQUIRES(--interact)
>>> # Interactive matplotlib stuff
>>> from xdev.interactive_iter import *  # NOQA
>>> import kwimage
>>> import kwplot
>>> plt = kwplot.autoplt(verbose=3, force='Qt5Agg')
>>> plt.ion()
>>> keys = list(kwimage.grab_test_image.keys())
>>> iterable = [kwimage.grab_test_image(key) for key in keys]
>>> iiter = InteractiveIter(iterable)
>>> for img in iiter:
>>>     kwplot.imshow(img)
>>>     InteractiveIter.draw()

Parameters:

iterable (None) – (default = None)
enabled (bool) – (default = True)
startx (int) – (default = 0)
default_action (str) – (default = ‘next’)
custom_actions (list) – list of 4-tuple (name, actions, help, func) (default = [])
wraparound (bool) – (default = False)
display_item (bool) – (default = True)
verbose (bool) – verbosity flag(default = True)

classmethod eventloop(custom_actions=[])[source]¶: For use outside of iteration wrapping. Makes an interactive event loop custom_actions should be specified in format [dispname, keys, desc, func]

handle_ans(ans_)[source]¶: preforms an actionm based on a user answer

prompt()[source]¶

wait_for_input()[source]¶

classmethod draw()[source]¶: in the common case where InteractiveIter is used to view matplotlib figures, you will have to draw the figure manually. This is a helper for that task.

class xdev.MultiPattern(patterns, predicate)[source]¶

Bases: PatternBase, NiceRepr

Example

>>> dpath = ub.Path.appdir('xdev/tests/multipattern_paths').ensuredir().delete().ensuredir()
>>> (dpath / 'file0.txt').touch()
>>> (dpath / 'data0.dat').touch()
>>> (dpath / 'other0.txt').touch()
>>> ((dpath / 'dir1').ensuredir() / 'file1.txt').touch()
>>> ((dpath / 'dir2').ensuredir() / 'file2.txt').touch()
>>> ((dpath / 'dir2').ensuredir() / 'file3.txt').touch()
>>> ((dpath / 'dir1').ensuredir() / 'data.dat').touch()
>>> ((dpath / 'dir2').ensuredir() / 'data.dat').touch()
>>> ((dpath / 'dir2').ensuredir() / 'data.dat').touch()
>>> pat = MultiPattern.coerce(['*.txt'], 'glob')
>>> print(list(pat.paths(cwd=dpath)))
>>> pat = MultiPattern.coerce(['*0*', '**/*.txt'], 'glob')
>>> print(list(pat.paths(cwd=dpath, recursive=1)))
>>> pat = MultiPattern.coerce(['*.txt', '**/*.txt', '**/*.dat'], 'glob')
>>> print(list(pat.paths(cwd=dpath)))

match(text)[source]¶

paths(cwd=None, recursive=False)[source]¶

_squeeze()[source]¶

classmethod coerce(data, hint='auto', predicate='any')[source]¶

Parameters:

data (str | List | Pattern | PathLike | MultiPattern)
hint (str) – can be ‘glob’, ‘regex’, ‘strict’ or ‘auto’. In ‘auto’ we will use ‘glob’ if the input is a string and ‘*’ is in the pattern, otherwise we will use strict. Pattern inputs keep their existing interpretation.

Returns:

MultiPattern

Example

>>> pat = MultiPattern.coerce('foo*', 'glob')
>>> pat2 = MultiPattern.coerce(pat, 'regex')
>>> pat3 = MultiPattern.coerce([pat, pat], 'regex')
>>> pat4 = MultiPattern.coerce([ub.Path('bar*'), pat], 'regex')
>>> print('pat = {}'.format(ub.urepr(pat, nl=1)))
>>> print('pat2 = {}'.format(ub.urepr(pat2, nl=1)))
>>> print('pat3 = {!r}'.format(pat3))
>>> print('pat4 = {!r}'.format(pat4))

>>> pat00 = MultiPattern.coerce('foo', 'glob')
>>> pat01 = MultiPattern.coerce('foo*', 'glob')
>>> pat02 = MultiPattern.coerce('foo*', 'regex')
>>> pat5 = MultiPattern.coerce(['foo', 'foo*', pat, pat00, pat01, pat02])
>>> print(f'pat5={pat5}')

Example

>>> # Test all acceptable input types
>>> import itertools as it
>>> str_pat = 'pattern*'
>>> scalar_inputs = {
>>>     'str': str_pat,
>>>     'path': ub.Path(str_pat),
>>>     'pat': Pattern.coerce(str_pat),
>>>     'mpat': MultiPattern.coerce(str_pat)
>>> }
>>> # Test scalar input types
>>> scalar_outputs = {}
>>> for k, v in scalar_inputs.items():
>>>     scalar_outputs[k] = MultiPattern.coerce(v)
>>> print('scalar_outputs = {}'.format(ub.urepr(scalar_outputs, nl=1)))
>>> #
>>> # Test iterable input types
>>> multi_outputs = []
>>> for v in it.combinations(scalar_inputs.values(), 2):
>>>     multi_outputs.append(MultiPattern.coerce(v))
>>> for v in it.combinations(scalar_inputs.values(), 3):
>>>     multi_outputs.append(MultiPattern.coerce(v))
>>> # Higher order nesting test
>>> higher_order_output = MultiPattern.coerce(multi_outputs)
>>> print('higher_order_output = {}'.format(ub.urepr(higher_order_output, nl=1)))

class xdev.Pattern(pattern, backend)[source]¶

Bases: PatternBase, NiceRepr

Provides a common API to several common pattern matching syntaxes.

A general patterns class, which can use a backend from BACKENDS

Parameters:

pattern (str | object) – The pattern text or a precompiled backend pattern object
backend (str) – Code indicating what backend the pattern text should be interpereted with. See BACKENDS for available choices.

Notes

# BACKENDS

The glob backend uses the fnmatch module [fnmatch_docs]. The regex backend uses the Python re module. The strict backend uses the “==” string equality testing. The parse backend uses the parse module.

References

[fnmatch_docs]

https://docs.python.org/3/library/fnmatch.html

Example

>>> # Test Regex backend
>>> repat = Pattern.coerce('foo.*', 'regex')
>>> assert repat.match('foobar')
>>> assert not repat.match('barfoo')
>>> match = repat.search('baz-biz-foobar')
>>> match = repat.match('baz-biz-foobar')
>>> # Test Glob backend
>>> globpat = Pattern.coerce('foo*', 'glob')
>>> assert globpat.match('foobar')
>>> assert not globpat.match('barfoo')
>>> globpat = Pattern.coerce('[foo|bar]', 'glob')
>>> globpat.match('foo')

Example

>>> # xdoctest: +REQUIRES(module:parse)
>>> # Test parse backend
>>> pattern1 = Pattern.coerce('A {adjective} pattern', 'parse')
>>> result1 = pattern1.match('A cool pattern')
>>> print(f'result1.named = {ub.urepr(result1.named, nl=1)}')
>>> pattern2 = pattern1.to_regex()
>>> result2 = pattern2.match('A cool pattern')

to_regex()[source]¶

Returns an equivalent pattern with the regular expression backend

Returns:: Pattern

Example

>>> globpat = Pattern.coerce('foo*', 'glob')
>>> strictpat = Pattern.coerce('foo*', 'strict')
>>> repat1 = strictpat.to_regex()
>>> repat2 = globpat.to_regex()
>>> print(f'repat1={repat1}')
>>> print(f'repat2={repat2}')

classmethod from_regex(data, flags=0, multiline=False, dotall=False, ignorecase=False)[source]¶: Create a Pattern object with a regex backend.

classmethod from_glob(data)[source]¶: Create a Pattern object with a glob backend.

classmethod coerce_backend(data, hint='auto')[source]¶

Example

>>> assert Pattern.coerce_backend('foo', hint='auto') == 'strict'
>>> assert Pattern.coerce_backend('foo*', hint='auto') == 'glob'
>>> assert Pattern.coerce_backend(re.compile('foo*'), hint='auto') == 'regex'

classmethod coerce(data, hint='auto')[source]¶

Attempt to automatically determine the input data as the appropriate pattern. If it cannot be determined, then fallback to the hint.

Parameters:

data (str | Pattern | PathLike)
hint (str) – can be ‘glob’, ‘regex’, ‘strict’ or ‘auto’. In ‘auto’ we will use ‘glob’ if the input is a string and ‘*’ is in the pattern, otherwise we will use strict. Pattern inputs keep their existing interpretation.

Example

>>> pat = Pattern.coerce('foo*', 'glob')
>>> pat2 = Pattern.coerce(pat, 'regex')
>>> print('pat = {}'.format(ub.urepr(pat, nl=1)))
>>> print('pat2 = {}'.format(ub.urepr(pat2, nl=1)))

match(text)[source]¶

search(text)[source]¶

sub(repl, text, count=-1)[source]¶

Parameters:

repl (str) – text to insert in place of pattern
text (str) – text to be searched and modified
count (int) – if non-negative, the maximum number of replacements that will be made.

paths(cwd=None, recursive=False)[source]¶

Find paths in the filesystem that match this pattern

Yields:: ub.Path

class xdev.PatternBase[source]¶

Bases: object

Abstract class that defines the Pattern api

match(text)[source]¶

search(text)[source]¶

sub(repl, text)[source]¶

class xdev.PythonRegexBuilder[source]¶

Bases: RegexBuilder

Contains helper methods to construct a regex

Example

>>> b = PythonRegexBuilder()
>>> pat_text = b.lookbehind('_') + r'v\d+' + b.optional(b.lookahead('_'))
>>> pat = re.compile(pat_text)
>>> print(pat.search('_v321_').group())
v321
>>> print(pat.search('_v321').group())
v321
>>> print(pat.search('fdsfds_v321_fdsfsd').group())
v321
>>> print(pat.search('fdsfds_v321fdsfsd').group())
v321
>>> print(pat.search('fdsfdsv321fdsfsd'))
None

Example

>>> # Test multiple negative lookbehind
>>> b = PythonRegexBuilder()
>>> suffix = 'foo'
>>> neg_prefix1 = b.lookbehind('abc', positive=0)
>>> neg_prefix2 = b.lookbehind('efg', positive=0)
>>> pat1 = re.compile(neg_prefix1 + suffix)
>>> pat2 = re.compile(neg_prefix2 + suffix)
>>> patB = re.compile(neg_prefix1 + neg_prefix2 + suffix)
>>> cases = ['abcfoo', 'efgfoo', 'hijfoo', 'foo']
>>> print([bool(pat1.search(c)) for c in cases])
>>> print([bool(pat2.search(c)) for c in cases])
>>> print([bool(patB.search(c)) for c in cases])
[False, True, True, True]
[True, False, True, True]
[False, False, True, True]

References

https://www.dataquest.io/blog/regex-cheatsheet/ https://docs.python.org/3/library/re.html#regular-expression-syntax

python_patterns = [{'alias': ['nongreedy_kleene_star'], 'docs': 'non-greedily matches zero or more of the pattern to the left', 'key': 'nongreedy_zero_or_more', 'pattern': '*?'}, {'docs': 'The boundary at the start or end of a word', 'key': 'boundary', 'pattern': '\\b'}, {'key': 'non-boundary', 'pattern': '\\B'}, {'key': 'left-expr', 'pattern': '\\A'}, {'docs': 'Matches only at the end of the string', 'key': 'right-expr', 'pattern': '\\Z'}]¶

previous(min=None, max=None, exact=None, greedy=True)[source]¶

Match the previous pattern some number of times.

Parameters:

min (int | None) – minimum number of matches
max (int | None) – maximum number of matches
exact (int | None) – Specify exact number of matches. Mutex with minimum and max.
greedy (bool) – if True match as many as possible, otherwise match as few as possible

Example

>>> from xdev.regex_builder import *  # NOQA
>>> b = PythonRegexBuilder()
>>> assert b.previous(exact=1) == '{1}'
>>> assert b.previous(min=1, max=3) == '{1,3}'
>>> assert b.previous(min=1, max=3, greedy=False) == '{1,3}?'
>>> assert b.previous(max=3) == '{,3}'
>>> assert b.previous(min=3) == '{3,}'
>>> assert b.previous() == '*'
>>> assert b.previous(greedy=False) == '*?'

Example

>>> from xdev.regex_builder import *  # NOQA
>>> b = PythonRegexBuilder()
>>> assert re.compile('a' + b.previous(exact=2) + '$').match('aa')
>>> assert not re.compile('a' + b.previous(exact=2) + '$').match('aaa')
>>> assert not re.compile('a' + b.previous(exact=2) + '$').match('a')

xdev.RE_Pattern¶: alias of Pattern

class xdev.RegexBuilder[source]¶

Bases: object

Notes

The way to have multiple negative look aheads/behinds is to change them together SO12689046

References

[SO12689046]

https://stackoverflow.com/questions/12689046/multiple-negative-lookbehind-assertions-in-python-regex

Example

b = RegexBuilder.coerce(‘python’) import re pat = re.compile(‘[A-Z-]+’)

common_patterns = [{'docs': 'An alphanumeric word, i.e. [a-zA-Z0-9_] (also matches unicode characters in Python)', 'key': 'word', 'pattern': '\\w'}, {'docs': 'Anything not a word', 'key': 'non-word', 'pattern': '\\W'}, {'docs': 'Any space character including: " " "\\t", "\\n", "\\r"', 'key': 'space', 'pattern': '\\s'}, {'docs': 'Any non-space character', 'key': 'non-space', 'pattern': '\\S'}, {'docs': 'any number 0-9', 'key': 'digit', 'pattern': '\\d'}, {'docs': 'any non-digit', 'key': 'digit', 'pattern': '\\D'}, {'alias': ['kleene_star'], 'docs': 'zero or more of the pattern to the left', 'key': 'zero_or_more', 'pattern': '*'}]¶

lookahead(pat, positive=True, mode='positive')[source]¶

A lookahead pattern that can be positive or negative

looklook

lookbehind(pat, positive=True)[source]¶: A lookbehind pattern that can be positive or negative

named_field(pat, name=None)[source]¶

bref_field(name)[source]¶

escape(pat)[source]¶

optional(pat)[source]¶

group(pat)[source]¶

oneof(*paterns)[source]¶

classmethod coerce(backend='python')[source]¶

property identifier¶

A word, except it must start with a letter or underscore (not a number)

References

https://stackoverflow.com/questions/5474008/regular-expression-to-confirm-whether-a-string-is-a-valid-python-identifier

Example

>>> from xdev.regex_builder import *  # NOQA
>>> b = PythonRegexBuilder()
>>> assert re.match(b.identifier, 'hello')
>>> assert re.match(b.identifier, 'hello')
>>> assert re.match(b.identifier, '𝛣_ello')
>>> assert re.match(b.identifier, 'h_1e8llo')
>>> assert not re.match(b.identifier, '1hello')

property hex¶: A case-independent hex character

property word¶

property whitespace¶

property nongreedy¶

property number¶

Can match a generic floating point number

References

https://www.regular-expressions.info/floatingpoint.html

Example

>>> from xdev.regex_builder import *  # NOQA
>>> b = PythonRegexBuilder()
>>> pat = re.compile('^' + b.number + '$')
>>> assert pat.match('3.4')
>>> assert pat.match('3.4e-1')
>>> assert pat.match('3.4')
>>> assert pat.match('3.4e+1')
>>> assert not pat.match('3.4a+1')

>>> b = PythonRegexBuilder()
>>> num_part = b.named_field(b.number, name='number')
>>> space_part = b.named_field(' *', name='spaces')
>>> unit_part = b.named_field('.*', name='unit')
>>> pat = re.compile('^' + num_part + space_part + unit_part + '$')
>>> pat.match('3.4').groupdict()
>>> pat.match('3.1415 foobars').groupdict()
>>> pat.match('3.1415foobars').groupdict()
>>> pat.match('+3.1415e9foobars').groupdict()

class xdev.UtfDirectedGlyphs[source]¶

Bases: _UtfBaseGlyphs

last = '└─╼ '¶

mid = '├─╼ '¶

backedge = '╾'¶

vertical_edge = '╽'¶

class xdev.UtfUndirectedGlyphs[source]¶

Bases: _UtfBaseGlyphs

last = '└── '¶

mid = '├── '¶

backedge = '─'¶

vertical_edge = '│'¶

class xdev.VimRegexBuilder[source]¶

Bases: RegexBuilder

https://dev.to/iggredible/learning-vim-regex-26ep

vim_patterns = [{'alias': ['nongreedy_kleene_star'], 'docs': 'non-greedily matches zero or more of the pattern to the left', 'key': 'nongreedy_zero_or_more', 'pattern': '\\{-}'}]¶

previous(min=None, max=None, exact=None, greedy=True)[source]¶

Match the previous pattern some number of times.

Parameters:

min (int | None) – minimum number of matches
max (int | None) – maximum number of matches
exact (int | None) – Specify exact number of matches. Mutex with minimum and max.
greedy (bool) – if True match as many as possible, otherwise match as few as possible

Example

>>> from xdev.regex_builder import *  # NOQA
>>> b = VimRegexBuilder()
>>> assert b.previous(exact=1) == r'\{1}'
>>> assert b.previous(min=1, max=3) == r'\{1,3}'
>>> assert b.previous(min=1, max=3, greedy=False) == r'\{-1,3}'
>>> assert b.previous(max=3) == r'\{,3}'
>>> assert b.previous(min=3) == r'\{3,}'
>>> assert b.previous() == '*'
>>> assert b.previous(greedy=False) == r'\{-}'

xdev.bubbletext(text, font='cybermedium')[source]¶

Uses pyfiglet to create bubble text.

Parameters:: font (str) – default=cybermedium, other fonts include: cybersmall and cyberlarge.

References

http://www.figlet.org/

Example

>>> bubble_text = bubbletext('TESTING BUBBLE TEXT', font='cybermedium')
>>> print(bubble_text)

xdev.byte_str(num, unit='auto', precision=2)[source]¶

Automatically chooses relevant unit (KB, MB, or GB) for displaying some number of bytes.

Parameters:

num (int) – number of bytes
unit (str) – which unit to use, can be auto, B, KB, MB, GB, TB, PB, EB, ZB, or YB.
precision (int) – number of decimals of precision

References

https://en.wikipedia.org/wiki/Orders_of_magnitude_(data)

Returns:: string representing the number of bytes with appropriate units
Return type:: str

Example

>>> num_list = [1, 100, 1024,  1048576, 1073741824, 1099511627776]
>>> result = ub.repr2(list(map(byte_str, num_list)), nl=0)
>>> print(result)
['0.00 KB', '0.10 KB', '1.00 KB', '1.00 MB', '1.00 GB', '1.00 TB']

xdev.conj_phrase(list_, cond='or')[source]¶

Joins a list of words using English conjunction rules

Parameters:

list_ (list) – of strings
cond (str) – a conjunction (or, and, but)

Returns:

the joined cconjunction phrase

Return type:

str

References

http://en.wikipedia.org/wiki/Conjunction_(grammar)

Example

>>> list_ = ['a', 'b', 'c']
>>> result = conj_phrase(list_, 'or')
>>> print(result)
a, b, or c

Example1:

>>> list_ = ['a', 'b']
>>> result = conj_phrase(list_, 'and')
>>> print(result)
a and b

xdev.difftext(text1, text2, context_lines=0, ignore_whitespace=False, colored=False)[source]¶

Uses difflib to return a difference string between two similar texts

Parameters:

text1 (str) – old text
text2 (str) – new text
context_lines (int) – number of lines of unchanged context
ignore_whitespace (bool)
colored (bool) – if true highlight the diff

Returns:

formatted difference text message

Return type:

str

References

http://www.java2s.com/Code/Python/Utility/IntelligentdiffbetweentextfilesTimPeters.htm

Example

>>> # build test data
>>> text1 = 'one\ntwo\nthree'
>>> text2 = 'one\ntwo\nfive'
>>> # execute function
>>> result = difftext(text1, text2)
>>> # verify results
>>> print(result)
- three
+ five

Example

>>> # build test data
>>> text1 = 'one\ntwo\nthree\n3.1\n3.14\n3.1415\npi\n3.4\n3.5\n4'
>>> text2 = 'one\ntwo\nfive\n3.1\n3.14\n3.1415\npi\n3.4\n4'
>>> # execute function
>>> context_lines = 1
>>> result = difftext(text1, text2, context_lines, colored=True)
>>> # verify results
>>> print(result)

xdev.distext(obj)[source]¶

Like dis.dis, but returns the text

Parameters:: obj – a compiled object (e.g. a function, class, generator, etc…)
Returns:: text of the python byte code
Return type:: str

Example

>>> from xdev.introspect import *  # NOQA
>>> import ubelt as ub
>>> code = ub.codeblock(
    '''
    def foo(a, b):
        print('hello')
        if __debug__:
            if a is None or b is None:
                raise ValueError
        print('world')
        # This is not entirely optimized away
        if __debug__ and (a is None or b is None):
            raise ValueError
        return a + b
    ''')
>>> obj = compile(code, filename='<memory>', mode='exec', dont_inherit=True, optimize=0)
>>> print(' ** UNOPTIMIZED')
>>> text = distext(obj)
>>> print(text)
>>> print(' ** OPTIMIZED')
>>> obj = compile(code, filename='<memory>', mode='exec', dont_inherit=True, optimize=1)
>>> text = distext(obj)
>>> print(text)

xdev.edit_distance(string1, string2)[source]¶

Edit distance algorithm. String1 and string2 can be either strings or lists of strings

Parameters:

string1 (str | List[str])
string2 (str | List[str])

Requirements:: pip install python-Levenshtein

Returns:: float | List[float] | List[List[float]]

Example

>>> # xdoctest: +REQUIRES(module:Levenshtein)
>>> string1 = 'hello world'
>>> string2 = ['goodbye world', 'rofl', 'hello', 'world', 'lowo']
>>> edit_distance(['hello', 'one'], ['goodbye', 'two'])
>>> edit_distance('hello', ['goodbye', 'two'])
>>> edit_distance(['hello', 'one'], 'goodbye')
>>> edit_distance('hello', 'goodbye')
>>> distmat = edit_distance(string1, string2)
>>> result = ('distmat = %s' % (ub.repr2(distmat),))
>>> print(result)
>>> [7, 9, 6, 6, 7]

xdev.editfile(fpath, verbose=True)[source]¶

Opens a file or code corresponding to a live python object in your preferred visual editor. This function is mainly useful in an interactive IPython session.

The visual editor is determined by the VISUAL environment variable. If this is not specified it defaults to gvim.

Parameters:

fpath (PathLike | ModuleType | str) – a file path or python module / function. If the input is a string it will interpret it either as a Path or a module name. Ambiguity is resolved by choosing a path if the string resolves to an existing path, and then checking if the string corresponds to a module name.
verbose (int) – verbosity

Example

>>> # xdoctest: +SKIP
>>> # This test interacts with a GUI frontend, not sure how to test.
>>> import xdev
>>> ub.editfile(xdev.misc.__file__)
>>> ub.editfile(xdev)
>>> ub.editfile(xdev.editfile)

xdev.embed(parent_locals=None, parent_globals=None, exec_lines=None, remove_pyqt_hook=True, n=0)[source]¶

Starts interactive session. Similar to keyboard command in matlab. Wrapper around IPython.embed.

Note

Contains helper logic to allow the developer to more easilly exit the program if embed is called in a loop. Specifically if you want to quit and not embed again, then set qqq=1 before exiting the embed session.

SeeAlso:: embed_on_exception()

xdev.embed_if_requested(n=0)[source]¶

Calls xdev.embed conditionally based on the environment.

Useful in cases where you want to leave the embed call around, but you dont want it to trigger in normal circumstances.

Specifically, embed is only called if the environment variable XDEV_EMBED exists or if –xdev-embed is in sys.argv.

xdev.find(pattern=None, dpath=None, include=None, exclude=None, dirblocklist=None, type=None, recursive=True, followlinks=False)[source]¶

Find all paths in a root subject to a search criterion

Parameters:

pattern (str | Pattern | None) – The glob pattern the path name must match to be returned
dpath (str | Pattern | None) – The root directory to search. Can also be a filepath, in which case, that is the only filepath considered. NOTE: in the future, this argument may change to path to indicate specifying a filepath is allowed. Defaults to cwd.
include (str | List[str] | MultiPattern | None) – Pattern or list of patterns. If specified, search only files whose base name matches this pattern. By default the pattern is GLOB. This only applies to the final name. Directories that do not match this name will still be traversed.
exclude (str | List[str] | MultiPattern | None) – Pattern or list of patterns. Skip any file with a name suffix that matches the pattern. By default the pattern is GLOB. This ONLY applies to the final name. Directories that match an exclude pattern will still be traversed. Use dirblocklist to specify patterns to exclude intermediate directories from traversal.
dirblocklist (str | List[str] | MultiPattern | None) – Any directory name matching this pattern will be removed from traversal.
type (str | List[str] | None) – A list of 1 character codes indicating what types of file can be returned. Currently we only allow either “f” for file or “d” for directory. Symbolic links are not currently distinguished. In the future we may support posix codes, see [1]_ for details.
recursive – search all subdirectories recursively
followlinks (bool, default=False) – if True will follow directory symlinks

References

_[1] https://linuxconfig.org/identifying-file-types-in-linux

Todo

mindepth

maxdepth

ignore_case

regex_match

Example

>>> from xdev.search_replace import *  # NOQA
>>> from xdev.search_replace import _create_test_filesystem
>>> dpath = _create_test_filesystem()['root']
>>> paths = list(find(pattern='*', dpath=dpath))
>>> assert len(paths) == 5
>>> paths = list(find(pattern='*', dpath=dpath, type='f'))
>>> assert len(paths) == 4

xdev.fix_embed_globals()[source]¶

HACK adds current locals() to globals(). Can be dangerous.

References

https://github.com/ipython/ipython/issues/62

Solves the following issue:

def foo():
x = 5 # You embed here import xdev xdev.embed()

‘’’ Now you try and run this line manually but you get a NameError

result = [x + i for i in range(10)]

No problem, just use. It changes all local variables to globals xdev.fix_embed_globals()

‘’’ result = [x + i for i in range(10)]

foo()

xdev.format_quotes_in_file(fpath, diff=True, write=False, verbose=3)[source]¶

Autoformat quotation marks in Python files

Parameters:

fpath (str) – The file to format
diff (bool) – if True write the diff between old and new to stdout
write (bool) – if True write the modifications to disk
verbose (int) – verbosity level

xdev.format_quotes_in_text(text, backend='parso')[source]¶

Reformat text according to formatting rules

Parameters:: text (str) – python source code
Returns:: modified text
Return type:: str

Example

>>> # xdoctest: +REQUIRES(module:parso)
>>> from xdev.format_quotes import *  # NOQA
>>> text = ub.codeblock(
...     f'''
...     def func1():
...         {TRIPLE_SINGLE_QUOTE}
...         Fix this.
...         {TRIPLE_SINGLE_QUOTE}
...
...     def func2():
...         {TRIPLE_DOUBLE_QUOTE}
...         Leave the doctests alone!
...         {TRIPLE_DOUBLE_QUOTE}
...
...     string1 = "fix these"
...     string2 = "don't fix these"
...     string3 = {TRIPLE_SINGLE_QUOTE}this is ok{TRIPLE_SINGLE_QUOTE}
...     string4 = {TRIPLE_DOUBLE_QUOTE}fix this{TRIPLE_DOUBLE_QUOTE}
...
...     def func3():
...         inside_string1 = "fix these"
...         inside_string2 = "don't fix these"
...         inside_string3 = {TRIPLE_SINGLE_QUOTE}this is ok{TRIPLE_SINGLE_QUOTE}
...         inside_string4 = {TRIPLE_DOUBLE_QUOTE}fix this{TRIPLE_DOUBLE_QUOTE}
...     ''')
>>> print(text)
>>> fixed = format_quotes_in_text(text)
>>> print(fixed)
>>> import xdev
>>> fixed = format_quotes_in_text(text, backend='parso')
>>> print('----')
>>> print(xdev.difftext(text, fixed, colored=True))
>>> # xdoctest: +REQUIRES(module:redbaron)
>>> print('----')
>>> fixed = format_quotes_in_text(text, backend='redbaron')
>>> print('----')
>>> print(xdev.difftext(text, fixed, colored=True))
>>> print('----')

xdev.generate_network_text(graph, with_labels=True, sources=None, max_depth=None, ascii_only=False, vertical_chains=False)[source]¶

Generate lines in the “network text” format

This works via a depth-first traversal of the graph and writing a line for each unique node encountered. Non-tree edges are written to the right of each node, and connection to a non-tree edge is indicated with an ellipsis. This representation works best when the input graph is a forest, but any graph can be represented.

This notation is original to networkx, although it is simple enough that it may be known in existing literature. See #5602 for details. The procedure is summarized as follows:

1. Given a set of source nodes (which can be specified, or automatically discovered via finding the (strongly) connected components and choosing one node with minimum degree from each), we traverse the graph in depth first order.

Each reachable node will be printed exactly once on it’s own line.
Edges are indicated in one of three ways:

a. a parent “L-style” connection on the upper left. This corresponds to a traversal in the directed DFS tree.

b. a backref “<-style” connection shown directly on the right. For directed graphs, these are drawn for any incoming edges to a node that is not a parent edge. For undirected graphs, these are drawn for only the non-parent edges that have already been represented (The edges that have not been represented will be handled in the recursive case).

c. a child “L-style” connection on the lower right. Drawing of the children are handled recursively.

4. The children of each node (wrt the directed DFS tree) are drawn underneath and to the right of it. In the case that a child node has already been drawn the connection is replaced with an ellipsis (”…”) to indicate that there is one or more connections represented elsewhere.

5. If a maximum depth is specified, an edge to nodes past this maximum depth will be represented by an ellipsis.

Parameters:

graph (nx.DiGraph | nx.Graph) – Graph to represent
with_labels (bool | str) – If True will use the “label” attribute of a node to display if it exists otherwise it will use the node value itself. If given as a string, then that attribte name will be used instead of “label”. Defaults to True.
sources (List) – Specifies which nodes to start traversal from. Note: nodes that are not reachable from one of these sources may not be shown. If unspecified, the minimal set of nodes needed to reach all others will be used.
max_depth (int | None) – The maximum depth to traverse before stopping. Defaults to None.
ascii_only (Boolean) – If True only ASCII characters are used to construct the visualization
vertical_chains (Boolean) – If True, chains of nodes will be drawn vertically when possible.

Yields:

str (a line of generated text)
Example – >>> # xdoctest: +REQUIRES(module:networkx) >>> graph = nx.path_graph(10) >>> graph.add_node(‘A’) >>> graph.add_node(‘B’) >>> graph.add_node(‘C’) >>> graph.add_node(‘D’) >>> graph.add_edge(9, ‘A’) >>> graph.add_edge(9, ‘B’) >>> graph.add_edge(9, ‘C’) >>> graph.add_edge(‘C’, ‘D’) >>> graph.add_edge(‘C’, ‘E’) >>> graph.add_edge(‘C’, ‘F’) >>> write_network_text(graph) ╙── 0

└── 1

└── 2

└── 3

└── 4

└── 5

└── 6

└── 7

└── 8

└── 9
├── A ├── B └── C

├── D ├── E └── F
```
>>> write_network_text(graph, vertical_chains=True)
╙── 0
    │
    1
    │
    2
    │
    3
    │
    4
    │
    5
    │
    6
    │
    7
    │
    8
    │
    9
    ├── A
    ├── B
    └── C
        ├── D
        ├── E
        └── F
```

xdev.get_func_kwargs(func, max_depth=None)[source]¶

Dynamically parse the kwargs accepted by this function.

This function uses Python signatures where possible, but it also uses heuristics by inspecting the way any keywords dictionary is used.

Parameters:

func (callable) – function to introspect kwargs from
max_depth (int, default=None) – by default we recursively parse any kwargs passed to subfunctions.

xdev.get_stack_frame(N=0, strict=True)[source]¶

Parameters:

N (int) – N=0 means the frame you called this function in. N=1 is the parent frame.
strict (bool) – (default = True)

xdev.graph_str(graph, with_labels=True, sources=None, write=None, ascii_only=False)[source]¶

Creates a nice utf8 representation of a forest

This function has been superseded by nx.readwrite.text.generate_network_text(), which should be used instead.

Parameters:

graph (nx.DiGraph | nx.Graph) – Graph to represent (must be a tree, forest, or the empty graph)
with_labels (bool) – If True will use the “label” attribute of a node to display if it exists otherwise it will use the node value itself. Defaults to True.
sources (List) – Mainly relevant for undirected forests, specifies which nodes to list first. If unspecified the root nodes of each tree will be used for directed forests; for undirected forests this defaults to the nodes with the smallest degree.
write (callable) – Function to use to write to, if None new lines are appended to a list and returned. If set to the print function, lines will be written to stdout as they are generated. If specified, this function will return None. Defaults to None.
ascii_only (Boolean) – If True only ASCII characters are used to construct the visualization

Returns:

utf8 representation of the tree / forest

Return type:

str | None

Example

>>> # xdoctest: +REQUIRES(module:networkx)
>>> graph = nx.balanced_tree(r=2, h=3, create_using=nx.DiGraph)
>>> print(graph_str(graph))
╙── 0
    ├─╼ 1
    │   ├─╼ 3
    │   │   ├─╼ 7
    │   │   └─╼ 8
    │   └─╼ 4
    │       ├─╼ 9
    │       └─╼ 10
    └─╼ 2
        ├─╼ 5
        │   ├─╼ 11
        │   └─╼ 12
        └─╼ 6
            ├─╼ 13
            └─╼ 14

>>> # xdoctest: +REQUIRES(module:networkx)
>>> graph = nx.balanced_tree(r=1, h=2, create_using=nx.Graph)
>>> print(graph_str(graph))
╙── 0
    └── 1
        └── 2

>>> # xdoctest: +REQUIRES(module:networkx)
>>> print(graph_str(graph, ascii_only=True))
+-- 0
    L-- 1
        L-- 2

xdev.grep(regexpr, dpath=None, include=None, exclude=None, recursive=True, dirblocklist=None, verbose=1)[source]¶

Execute a grep on multiple files.

Parameters:

regexpr (str | Pattern) – pattern to find
dpath (str | None) – passed to find().
include (str | List[str] | MultiPattern | None) – passed to find().
exclude (str | List[str] | MultiPattern | None) – passed to find().
recursive (bool) – passed to find().
dirblocklist (str | List[str] | MultiPattern | None) – passed to find().
verbose (int) – verbosity level

Return type:

List[GrepResult]

Example

>>> from xdev.search_replace import *  # NOQA
>>> from xdev.search_replace import _create_test_filesystem
>>> dpath = _create_test_filesystem()['root']
>>> grep('a', dpath=dpath)

xdev.grepfile(fpath, regexpr, verbose=1)[source]¶

Exceute grep on a single file

Parameters:

fpath (str | PathLike) – file to search
regexpr (str | Pattern) – pattern to find
verbose (int) – verbosity level

Returns:

None | GrepResult

Example

>>> from xdev.search_replace import *  # NOQA
>>> from xdev.search_replace import _create_test_filesystem
>>> fpath = _create_test_filesystem()['contents'][1]
>>> grep_result = grepfile(fpath, r'\bb\b')
>>> print('grep_result = {}'.format(grep_result))

xdev.greptext(text, regexpr, fpath=None, verbose=1)[source]¶

Exceute grep on text

Parameters:

text (str) – text to search
regexpr (str | Pattern) – pattern to find
verbose (int) – verbosity level

Returns:

None | GrepResult

xdev.import_module_from_pyx(fname, dpath=None, error='raise', autojit=True, verbose=1, recompile=False, annotate=False)[source]¶

Attempts to import a module corresponding to a pyx file.

If the corresponding compiled module is not found, this can attempt to JIT-cythonize the pyx file.

Parameters:

fname (str) – The basename of the cython pyx file
dpath (str) – The directory containing the cython pyx file
error (str) – Can be “raise” or “ignore”
autojit (bool) – If True, we will cythonize and compile the pyx file if possible.
verbose (int) – verbosity level (higher is more verbose)
recompile (bool) – if True force recompile

Returns:

ModuleType | None – Returns the compiled and imported module if possible, otherwise None

Return type:

module

xdev.iter_object_tree(obj)[source]¶

xdev.knapsack(items, maxweight, method='iterative')[source]¶

Solve the knapsack problem by finding the most valuable subsequence of items subject that weighs no more than maxweight.

Parameters:

items (tuple) – is a sequence of tuples (value, weight, id_), where value is a number and weight is a non-negative integer, and id_ is an item identifier.
maxweight (numbers.Real) – a non-negative number indicating the maximum weight of items we can take.

Returns:

(total_value, items_subset) - a pair whose first element is the: sum of values in the most valuable subsequence, and whose second element is the subsequence. Subset may be different depending on implementation (ie top-odwn recusrive vs bottom-up iterative)

Return type:

tuple

References

http://codereview.stackexchange.com/questions/20569/dynamic-programming-solution-to-knapsack-problem http://stackoverflow.com/questions/141779/solving-the-np-complete-problem-in-xkcd http://www.es.ele.tue.nl/education/5MC10/Solutions/knapsack.pdf

Example

>>> # ENABLE_DOCTEST
>>> import ubelt as ub
>>> items = [(4, 12, 0), (2, 1, 1), (6, 4, 2), (1, 1, 3), (2, 2, 4)]
>>> maxweight = 15
>>> total_value1, items_subset1 = knapsack(items, maxweight, method='iterative')
>>> print('items_subset1 = {}'.format(ub.repr2(items_subset1, nl=1)))
>>> print('total_value1 = {}'.format(ub.repr2(total_value1, nl=1)))

Example

>>> # ENABLE_DOCTEST
>>> # Solve https://xkcd.com/287/
>>> weights = [2.15, 2.75, 3.35, 3.55, 4.2, 5.8] * 2
>>> items = [(w, w, i) for i, w in enumerate(weights)]
>>> maxweight = 15.05
>>> total_value1, items_subset1 = knapsack(items, maxweight, method='iterative')
>>> total_weight = sum([t[1] for t in items_subset1])
>>> print('total_weight = %r' % (total_weight,))
>>> print('items_subset1 = %r' % (items_subset1,))
>>> #assert items_subset1 == items_subset, 'NOT EQ\n%r !=\n%r' % (items_subset1, items_subset)

Timeit:

# >>> import ubelt as ub # >>> setup = ub.codeblock( # >>> ‘’’ # import ubelt as ut # weights = [215, 275, 335, 355, 42, 58] * 40 # items = [(w, w, i) for i, w in enumerate(weights)] # maxweight = 2505 # #import numba # #knapsack_numba = numba.autojit(knapsack_iterative) # #knapsack_numba = numba.autojit(knapsack_iterative_numpy) # ‘’’) # >>> # Test load time # >>> stmt_list1 = ub.codeblock( # >>> ‘’’ # knapsack_iterative(items, maxweight) # knapsack_ilp(items, maxweight) # #knapsack_numba(items, maxweight) # #knapsack_iterative_numpy(items, maxweight) # ‘’’).split(’n’)

# ut.util_dev.timeit_compare(stmt_list1, setup, int(5))

from xdev.algo import * # NOQA def knapsack_inputs(n):

rng = np.random items = [

(rng.randint(0, 10), rng.randint(0, 10), idx) for idx in range(n)]

return {‘items’: items, ‘maxweight’: 100}

def input_wrapper(func):

import functools @functools.wraps(func) def _wrap(kwargs):

return func(**kwargs)

return _wrap

import perfplot perfplot.show(

setup=knapsack_inputs, kernels=list(map(input_wrapper, [knapsack_iterative, knapsack_ilp])), n_range=[2 ** k for k in range(15)], xlabel=”maxweight”, equality_check=None,

)

xdev.knapsack_greedy(items, maxweight)[source]¶

non-optimal greedy version of knapsack algorithm does not sort input. Sort the input by largest value first if desired.

Parameters:

`items` (tuple) – is a sequence of tuples (value, weight, id_), where value is a scalar and weight is a non-negative integer, and id_ is an item identifier.
`maxweight` (scalar) – is a non-negative integer.

Example

>>> # ENABLE_DOCTEST
>>> items = [(4, 12, 0), (2, 1, 1), (6, 4, 2), (1, 1, 3), (2, 2, 4)]
>>> maxweight = 15
>>> total_value, items_subset = knapsack_greedy(items, maxweight)
>>> result =  'total_value = %r\n' % (total_value,)
>>> result += 'items_subset = %r' % (items_subset,)
>>> print(result)
total_value = 7
items_subset = [(4, 12, 0), (2, 1, 1), (1, 1, 3)]

xdev.knapsack_ilp(items, maxweight, verbose=False)[source]¶

solves knapsack using an integer linear program

Example

>>> # DISABLE_DOCTEST
>>> import ubelt as ub
>>> # Solve https://xkcd.com/287/
>>> weights = [2.15, 2.75, 3.35, 3.55, 4.2, 5.8, 6.55]
>>> values  = [2.15, 2.75, 3.35, 3.55, 4.2, 5.8, 6.55]
>>> indices = ['mixed fruit', 'french fries', 'side salad',
>>>            'hot wings', 'mozzarella sticks', 'sampler plate',
>>>            'barbecue']
>>> items = [(v, w, i) for v, w, i in zip(values, weights, indices)]
>>> #items += [(3.95, 3.95, 'mystery plate')]
>>> maxweight = 15.05
>>> verbose = True
>>> total_value, items_subset = knapsack_ilp(items, maxweight, verbose)
>>> print('items_subset = %s' % (ub.repr2(items_subset, nl=1),))

xdev.knapsack_iterative(items, maxweight)[source]¶

xdev.knapsack_iterative_int(items, maxweight)[source]¶

Iterative knapsack method

Math:: maximize sum_{i in T} v_i subject to sum_{i in T} w_i leq W

Notes

dpmat is the dynamic programming memoization matrix. dpmat[i, w] is the total value of the items with weight at most W T is idx_subset, the set of indicies in the optimal solution

Example

>>> # ENABLE_DOCTEST
>>> weights = [1, 3, 3, 5, 2, 1] * 2
>>> items = [(w, w, i) for i, w in enumerate(weights)]
>>> maxweight = 10
>>> items = [(.8, 700, 0)]
>>> maxweight = 2000
>>> print('maxweight = %r' % (maxweight,))
>>> print('items = %r' % (items,))
>>> total_value, items_subset = knapsack_iterative_int(items, maxweight)
>>> total_weight = sum([t[1] for t in items_subset])
>>> print('total_weight = %r' % (total_weight,))
>>> print('items_subset = %r' % (items_subset,))
>>> result =  'total_value = %.2f' % (total_value,)
>>> print(result)
total_value = 0.80

xdev.knapsack_iterative_numpy(items, maxweight)[source]¶

Iterative knapsack method

maximize sum_{i in T} v_i subject to sum_{i in T} w_i leq W

Notes

dpmat is the dynamic programming memoization matrix. dpmat[i, w] is the total value of the items with weight at most W T is the set of indicies in the optimal solution

xdev.make_warnings_print_tracebacks()[source]¶

Makes warnings show tracebacks when displayed.

Applies a monkeypatch to warnings.formatwarning that includes a traceback in the displayed warning message.

xdev.nested_type(obj, unions=False)[source]¶

Compute the :module:`typing` compatible annotation type.

Parameters:

obj (Any) – a typing template based on a specific object
unions (bool) – if True use unions, otherwise use Any

Returns:

type code (might change to return actual type)

Return type:

str

Example

>>> obj = {'a': [1, 2], 'b': [3, 4, 5]}
>>> print(nested_type(obj))
Dict[str, List[int]]

>>> import numpy as np
>>> obj = {'b': {'a': 1.0, 'b': 'foo', 'c': np.array([1, 2])}}
>>> print(nested_type(obj, unions=True))
Dict[str, Dict[str, float | ndarray | str]]

xdev.number_of_decimals(num)[source]¶

Parameters:: num (float)

References

stackoverflow.com/questions/6189956/finding-decimal-places

Example

>>> # ENABLE_DOCTEST
>>> num = 15.05
>>> result = number_of_decimals(num)
>>> print(result)
2

xdev.profile_globals()[source]¶: Adds the profile decorator to all global functions

xdev.profile_now(func)[source]¶

Wrap a function to print profile information after it is called.

Parameters:: func (Callable) – function to profile
Returns:: the wrapped function
Return type:: Callable

Example

>>> # xdoctest: +SKIP
>>> from xdev.profiler import *  # NOQA
>>> def func_to_profile():
>>>     list(range(10))
>>>     tuple(range(100))
>>>     set(range(1000))
>>> profile_now(func_to_profile)()  # xdoctest: +IGNORE_WANT

Timer unit: 1e-09 s

Total time: 2.7767e-05 s File: <ipython-input-11-049a3440df03> Function: func_to_profile at line 3

Line # Hits Time Per Hit % Time Line Contents¶

3 def func_to_profile(): 4 1 3200.0 3200.0 11.5 list(range(10)) 5 1 1949.0 1949.0 7.0 tuple(range(100)) 6 1 22618.0 22618.0 81.5 set(range(1000))

xdev.quantum_random(pure=False)[source]¶

Returns a quantum random number as a 32 bit unsigned integer. Does this by making a network request to the ANU Quantum Random Number Generator web service, so an internet connection is required.

Parameters:: pure (bool) – if False, mixes this data with pseudorandom data for security. Otherwise returns the raw quantum numbers that were sent over the web (i.e. subject to MitM attacks).

Requirements:: quantumrandom >= 1.9.0

Returns:: the random number
Return type:: numpy.uint32

xdev.reload_class(self, verbose=True, reload_module=True)[source]¶

Reload a class for a specific instance.

(populates any new methods that may have just been written)

Ported from utool

xdev.sed(regexpr, repl, dpath=None, include=None, exclude=None, dirblocklist=None, recursive=True, dry=False, verbose=1)[source]¶

Execute a sed on multiple files.

Parameters:

regexpr (str | Pattern) – pattern to find
repl (str) – the text to replace the found pattern with
dpath (str | None) – passed to find().
include (str | List[str] | MultiPattern | None) – passed to find().
exclude (str | List[str] | MultiPattern | None) – passed to find().
dirblocklist (str | List[str] | MultiPattern | None) – passed to find().
recursive (bool) – passed to find().
dry (bool) – if True does not apply edits
verbose (int) – verbosity level

Example

>>> from xdev.search_replace import *  # NOQA
>>> from xdev.search_replace import _create_test_filesystem
>>> dpath = _create_test_filesystem()['root']
>>> sed('a', 'x', dpath=dpath, dry=True)

xdev.sedfile(fpath, regexpr, repl, dry=False, verbose=1)[source]¶

Execute a search and replace on a particular file

Parameters:

fpath (str | PathLike) – file to search / replace on
regexpr (str | Pattern) – pattern to find
repl (str) – the text to replace the found pattern with
dry (bool) – if True does not apply edits
verbose (int) – verbosity level

Returns:

changed lines

Return type:

List[Tuple[str, str]]

Todo

[ ] Store “SedResult” class, with lazy execution

Example

>>> from xdev.search_replace import *  # NOQA
>>> from xdev.search_replace import _create_test_filesystem
>>> fpath = _create_test_filesystem()['contents'][1]
>>> changed_lines1 = sedfile(fpath, 'a', 'x', dry=True, verbose=1)
>>> changed_lines2 = sedfile(fpath, 'a', 'x', dry=False, verbose=0)
>>> assert changed_lines2 == changed_lines1
>>> changed_lines3 = sedfile(fpath, 'a', 'x', dry=False, verbose=0)
>>> assert changed_lines3 != changed_lines2

xdev.set_overlaps(set1, set2, s1='s1', s2='s2')[source]¶

Return sizes about set overlaps

Parameters:

set1 (Iterable)
set2 (Iterable)
s1 (str) – name for set1
s2 (str) – name for set2

Returns:

sizes of sets intersections unions and differences

Return type:

Dict[str, int]

Notes

This function needs a rename. Possible candidates brainstorm:

set_analysis
set_binary_analysis
set_binary_describe
set_relationships
describe_sets
describe_relations
describe_set_relations
sets_summary
sumarize_sets
sumerset

xdev.sidecar_glob(main_pat, sidecar_ext, main_key='main', sidecar_key=None, recursive=0)[source]¶

Similar to a regular glob, but returns a dictionary with associated main-file / sidecar-file pairs.

Todo

add as a general option to Pattern.paths?

Parameters:: main_pat (str | PathLike) – glob pattern for the main non-sidecar file
Yields:: Dict[str, ub.Path | None]

Notes

A sidecar file is defined by the sidecar extension. We usually use this for .dvc sidecars.

When the pattern includes a .dvc suffix, the result will include those .dvc files and any matching main files they correspond to. Note: if you search for paths like foo_*.dvc this might skiped unstaged files. Therefore it is recommended to only include the .dvc suffix in the pattern ONLY if you do not want any unstaged files.

If you want both staged and unstaged files, ensure the pattern does not exclude objects without a .dvc suffix (i.e. don’t end the pattern with .dvc).

When the pattern does not include a .dvc suffix, we include all those files, for other files that exist by adding a .dvc suffix.

With the pattern matches both a dvc and non-dvc file, they are grouped together.

Example

>>> from xdev.util_path import *  # NOQA
>>> dpath = ub.Path.appdir('xdev/tests/sidecar_glob')
>>> dpath.delete().ensuredir()
>>> (dpath / 'file1').touch()
>>> (dpath / 'file1.ext').touch()
>>> (dpath / 'file1.ext.car').touch()
>>> (dpath / 'file2.ext').touch()
>>> (dpath / 'file3.ext.car').touch()
>>> (dpath / 'file4.car').touch()
>>> (dpath / 'file5').touch()
>>> (dpath / 'file6').touch()
>>> (dpath / 'file6.car').touch()
>>> (dpath / 'file7.bike').touch()
>>> def _handle_results(results):
...     results = list(results)
...     for row in results:
...         for k, v in row.items():
...             if v is not None:
...                 row[k] = v.relative_to(dpath)
...     print(ub.repr2(results, sv=1))
...     return results
>>> main_key = 'main',
>>> sidecar_key = '.car'
>>> sidecar_ext = '.car'
>>> main_pat = dpath / '*'
>>> _handle_results(sidecar_glob(main_pat, sidecar_ext))
>>> _handle_results(sidecar_glob(dpath / '*.ext', '.car'))
>>> _handle_results(sidecar_glob(dpath / '*.car', '.car'))
>>> _handle_results(sidecar_glob(dpath / 'file*.ext', '.car'))
>>> _handle_results(sidecar_glob(dpath / '*', '.ext'))

xdev.startfile(fpath, verbose=True)[source]¶

Uses default program defined by the system to open a file. This is done via os.startfile on windows, open on mac, and xdg-open on linux.

Parameters:

fpath (PathLike) – a file to open using the program associated with the files extension type.
verbose (int) – verbosity

References

http://stackoverflow.com/questions/2692873/quote-posix

Example

>>> # xdoctest: +SKIP
>>> # This test interacts with a GUI frontend, not sure how to test.
>>> import ubelt as ub
>>> base = ub.ensure_app_cache_dir('ubelt')
>>> fpath1 = join(base, 'test_open.txt')
>>> ub.touch(fpath1)
>>> proc = ub.startfile(fpath1)

xdev.take_column(list_, colx)[source]¶: iterator version of take_column

xdev.test_object_pickleability(obj)[source]¶

xdev.tree(path)[source]¶

Like os.walk but yields a flat list of file and directory paths

Parameters:: path (str | os.PathLike) – path to traverse
Yields:: str – path

Example

>>> import itertools as it
>>> import ubelt as ub
>>> path = ub.Path('.')
>>> gen = tree(path)
>>> results = list(it.islice(gen, 5))
>>> print('results = {}'.format(ub.repr2(results, nl=1)))

xdev.tree_repr(cwd=None, max_files=100, dirblocklist=None, show_nfiles='auto', return_text=False, return_tree=False, pathstyle='name', max_depth=None, with_type=False, abs_root_label=True, ignore_dotprefix=True, colors=False)[source]¶

Filesystem tree representation

Like the unix util tree, but allow writing numbers of files per directory when given -d option

Parameters:

cwd (None | str | PathLike) – directory to print
max_files (int | None) – maximum files to print before supressing a directory
pathstyle (str) – can be rel, name, or abs
return_tree (bool) – if True return the tree
return_text (bool) – if True return the text
maxdepth (int | None) – maximum depth to descend
abs_root_label (bool) – if True force the root to always be absolute
colors (bool) – if True use rich

SeeAlso:: xdev.tree - generator

xdev.view_directory(dpath=None, verbose=False)[source]¶

View a directory in the operating system file browser. Currently supports windows explorer, mac open, and linux nautlius.

Parameters:

dpath (PathLike | None) – directory name
verbose (bool) – verbosity

xdev.write_network_text(graph, path=None, with_labels=True, sources=None, max_depth=None, ascii_only=False, end='\n', vertical_chains=False)[source]¶

Creates a nice text representation of a graph

This works via a depth-first traversal of the graph and writing a line for each unique node encountered. Non-tree edges are written to the right of each node, and connection to a non-tree edge is indicated with an ellipsis. This representation works best when the input graph is a forest, but any graph can be represented.

Parameters:

graph (nx.DiGraph | nx.Graph) – Graph to represent
path (string or file or callable or None) – Filename or file handle for data output. if a function, then it will be called for each generated line. if None, this will default to “sys.stdout.write”
with_labels (bool | str) – If True will use the “label” attribute of a node to display if it exists otherwise it will use the node value itself. If given as a string, then that attribte name will be used instead of “label”. Defaults to True.
sources (List) – Specifies which nodes to start traversal from. Note: nodes that are not reachable from one of these sources may not be shown. If unspecified, the minimal set of nodes needed to reach all others will be used.
max_depth (int | None) – The maximum depth to traverse before stopping. Defaults to None.
ascii_only (Boolean) – If True only ASCII characters are used to construct the visualization
end (string) – The line ending characater
vertical_chains (Boolean) – If True, chains of nodes will be drawn vertically when possible.

Example

>>> # xdoctest: +REQUIRES(module:networkx)
>>> graph = nx.balanced_tree(r=2, h=2, create_using=nx.DiGraph)
>>> write_network_text(graph)
╙── 0
    ├─╼ 1
    │   ├─╼ 3
    │   └─╼ 4
    └─╼ 2
        ├─╼ 5
        └─╼ 6

>>> # A near tree with one non-tree edge
>>> graph.add_edge(5, 1)
>>> write_network_text(graph)
╙── 0
    ├─╼ 1 ╾ 5
    │   ├─╼ 3
    │   └─╼ 4
    └─╼ 2
        ├─╼ 5
        │   └─╼  ...
        └─╼ 6

>>> graph = nx.cycle_graph(5)
>>> write_network_text(graph)
╙── 0
    ├── 1
    │   └── 2
    │       └── 3
    │           └── 4 ─ 0
    └──  ...

>>> graph = nx.cycle_graph(5, nx.DiGraph)
>>> write_network_text(graph, vertical_chains=True)
╙── 0 ╾ 4
    ╽
    1
    ╽
    2
    ╽
    3
    ╽
    4
    └─╼  ...

>>> write_network_text(graph, vertical_chains=True, ascii_only=True)
+-- 0 <- 4
    v
    1
    v
    2
    v
    3
    v
    4
    L->  ...

>>> graph = nx.generators.barbell_graph(4, 2)
>>> write_network_text(graph, vertical_chains=False)
╙── 4
    ├── 5
    │   └── 6
    │       ├── 7
    │       │   ├── 8 ─ 6
    │       │   │   └── 9 ─ 6, 7
    │       │   └──  ...
    │       └──  ...
    └── 3
        ├── 0
        │   ├── 1 ─ 3
        │   │   └── 2 ─ 0, 3
        │   └──  ...
        └──  ...
>>> write_network_text(graph, vertical_chains=True)
╙── 4
    ├── 5
    │   │
    │   6
    │   ├── 7
    │   │   ├── 8 ─ 6
    │   │   │   │
    │   │   │   9 ─ 6, 7
    │   │   └──  ...
    │   └──  ...
    └── 3
        ├── 0
        │   ├── 1 ─ 3
        │   │   │
        │   │   2 ─ 0, 3
        │   └──  ...
        └──  ...

>>> graph = nx.complete_graph(5, create_using=nx.Graph)
>>> write_network_text(graph)
╙── 0
    ├── 1
    │   ├── 2 ─ 0
    │   │   ├── 3 ─ 0, 1
    │   │   │   └── 4 ─ 0, 1, 2
    │   │   └──  ...
    │   └──  ...
    └──  ...

>>> graph = nx.complete_graph(3, create_using=nx.DiGraph)
>>> write_network_text(graph)
╙── 0 ╾ 1, 2
    ├─╼ 1 ╾ 2
    │   ├─╼ 2 ╾ 0
    │   │   └─╼  ...
    │   └─╼  ...
    └─╼  ...

class xdev.DirectoryWalker(dpath, exclude_dnames=None, exclude_fnames=None, include_dnames=None, include_fnames=None, max_walk_depth=None, max_files=None, parse_content=False, show_progress=True, ignore_empty_dirs=False, **kwargs)[source]¶

Bases: object

Configurable directory walker that can explore a directory and report information about its contents in a concise manner.

Options will impact how long this process takes based on how much data / metadata we need to parse out of the filesystem.

Parameters:

dpath (str | PathLike) – the path to walk
exclude_dnames (Coercable[MultiPattern]) – blocks directory names matching this pattern
exclude_fnames (Coercable[MultiPattern]) – blocks file names matching this pattern
include_dnames (Coercable[MultiPattern]) – if specified, excludes directories that do NOT match this pattern.
include_fnames (Coercable[MultiPattern]) – if specified, excludes files that do NOT match this pattern.
max_files (None | int) – ignore all files in directories with more than this number.
max_walk_depth (None | int) – how far to recurse
parse_content (bool) – if True, include content analysis
**kwargs – passed to label options

write_network_text(**kwargs)[source]¶

write_report(**nxtxt_kwargs)[source]¶

build()[source]¶

_inplace_filter_dnames(dnames)[source]¶

_inplace_filter_fnames(fnames)[source]¶

_walk()[source]¶

property file_paths¶

property dir_paths¶

_accum_stats()[source]¶

_update_stats()[source]¶

_update_stats2()[source]¶

_parallel_process_files(func, desc=None, max_workers=8, mode='thread')[source]¶: Applies a function to every node.

_humanize_stats(stats, node_type, reduce_prefix=False)[source]¶

_find_duplicate_files()[source]¶

_update_path_metadata()[source]¶

_update_labels()[source]¶: Update how each node will be displayed

_sort()[source]¶