MythNetTV talk

I gave a MythNetTV talk at linux.conf.au yesterday, and it went remarkably well. Hardly any fruit was thrown. I’m quite please that so far two patches have been sent to me after the talk, and questions in the talk promoted a release of version 7.

I should share the following resources from the talk: the slide deck, my cheat sheet, the MythTV book chapter beta which I wrote while gathering my thoughts for the talk. The last one is probably the most useful, as its pretty much an expanded version of the talk.

A historical note from November 2020: the mythtvbook.com website is now offline.

MythNetTV release 7

This release is smaller than the last one, but has some bug fixes which folk might find handy. New things in this release:

    Better testing:

  • r232: Simple script to create the unit testing database
    New features:

  • r210: Allow users to override the default bittorrent upload rate
  • r212: Provide more sync information when running in verbose mode
  • r216: Add the markunread command
  • r218: Make only one download attempt for a given show per 24 hour period
  • r220: Change download interval to one hour, update last attempt time more frequently
  • r222: Add –oldestfirst and –newestfirst flags, which alter NextDownloads behavior
  • r224: Add flag which forces downloads to run, even if they’ve failed recently
  • r226: Don’t mark a bittorrent download as complete just because the torrent file has been completely downloaded
  • r234: Use reasonable defaults for mysql configuration if no config file can be found
  • r236: Use reasonable defaults for mysql configuration if no config file can be found
  • r238: Use reasonable defaults for mysql configuration if no config file can be found
    Bug fixes:

  • r206: Make the output of “video.py length” more friendly
  • r208: Release man pages as well
  • r214: Fix a bug introduced in r210 which broke bittorrent downloads
  • r228: Fix a typo in the recordings_tool help text
  • r230: Fix a typo in the recordings_tool help text
  • r240: Reduce size of proxy name in the usage table to deal with key length limitations in MySQL
  • r242: The schema for the channel table changed in MythTV 0.21
  • r244: Fix typo
  • r246: Suppress repeated warnings of DB default use
  • r248: Bug fixes for linux.conf.au 2009 talk
  • r250: Include test runner script in release, prepare ChangeLog for release

Please grab your copy here.

MythNetTV release 6

New things in this release:

    Better testing:

  • r114: Unit tests for video.py
  • r148: Updated unit test now that we create more tables
  • r151: Start work on syndication unit tests
  • r153: Unit test for bad syndication dates, as well as more flexibility in
    db connections needed for this unit test

  • r157: Nicer unit test failure output, and refactor GetVideoDir() into the
    utility module

  • r181: Slightly improve syndication unit tests
  • r183: Add flag parsing to unit tests, and fix some more flag name changes
    Better documentation:

  • r117: Added a man page for video.py — I’m not 100% happy with its name
    though
    Bug fixes:

  • r114: Fixed a bug where the new filename for video transcode could be the
    same as the input filename, resulting in video corruption. This was found
    with one of the new video.py unit tests

  • r116: The logic for the –prompt flag was the wrong way around. Fixed.
  • r119: Nicer download status messages
  • r121: Handle 404s in feed updates better
  • r129: Slight tweak to SVN submit script
  • r131: More accurate tracking of proxy usage (update during download,
    instead of just at the end)

  • r137: Proxy budget being exceeded doesn’t count as a failed download
    attempt

  • r143: Subscribe now renables inactive subscriptions
  • r146: Add support to decimals to utility byte printer, fix a bug in the
    check for video directories

  • r155: Have users send problems to the mailing list, instead of me
    personally

  • r161: Don’t throw exceptions for the videodir command line
  • r167, 169: Display friendly sizes in records_tool output
  • r171: Move verbose update arg into a flag
  • r173: Add “-vo null” to mplayer invocation per Ryan Lutz. This improves
    support on machines without X, and speeds up the identify operation

  • r175: Import patch from Thomas Mashos which fixes subscription removal,
    started work on syndication unit test improvements

  • r177: Fix character escaping bug in show subtitles during import
  • r179: Renamed –datadirdefault to –datadir. If set this will now change
    your data directory, regardless of if there was a previous value set.

  • r190: Recording_tool now prompts for deletes
  • r192: Improved explainvideodir output
  • r194: Don’t crash in explainvideodir if there is no video directory
  • r197: Order results by subtitle in recordings_tool output
    New features:

  • r115: Upgraded schema to version 15 to support http_proxies for
    subscriptions. Added http_proxy command line, which allows you to use HTTP
    proxies for specified URLs. Moved HTTP accesses to use the proxy code.

  • r127: Bump schema to v17, and add proxy use tracking including the
    “proxyusage” command

  • r133: Allow daily budgets for proxy usage
  • r115: Provide a user agent for HTTP requests, instead of just
    Python-urllib/1.17

  • r117: Users will now be prompted to subscribe to an announcements video
    feed. This will happen exactly once. This behavior may be disabled with
    the –nopromptforannounce command line flag.

  • r125: Add a full info dump command to video.py’s command line interface
  • r139: Bump schema to 19, and implement categories with the “category”
    command

  • r141: Implement recording group support, and clarify category support
  • r151: Implement a helper (recordings_tool) for handling the MythTV
    recordings table, this is useful for testing.

  • r159: Add videodir and explainvideodir debugging commands, and update man
    page

  • r163: Add title list feature to recordings_tool
  • r165: Include recording count in title list
  • r185: Add the resetattempts command
    Development changes:

  • r123: Added a submit script to automate putting the revision number into
    the ChangeLog

  • r135: Tweak to new ChangeLog auto logging formatting

Release 6 continues the tradition of better testing, improves the documentation (a little, there is more work to be done there), fixes a bunch of bugs, and implements some new features which will hopefully be useful to others. Please grab your copy here.

Please help me test trunk

I’m getting ready to do another MythNetTV release, as I’ve fixed quite a few things recently. I’d appreciate people testing the new code before I do an official release. You can get the current code like this:

    svn co http://www.stillhq.com/mythtv/mythnettv/svn
    

That will create a directory called mythnettv, with a subdirectory named trunk, which is the latest development version of the code. You should probably create that directory somewhere where you don’t mind a new directory being created.

This release has lots of small changes, which are listed in the ChangeLog which will be downloaded with the code.

Thanks.

Couldn’t determine the video directory?

A couple of people have contacted me in the last couple of days with an error where MythNetTV can’t determine the directory to put videos in. The error would look something like this:

    Importing data/tekzilla--0050--tehbunniez--hd.h264.mp4 Traceback (most recent call last): File "./mythnettv", line 453, in main(sys.argv) File "./mythnettv", line 212, in main mythnettvcore.DownloadAndImport(db, guid, out=out) File "/home/mythbox/Scripts/mythnettv/mythnettvcore.py", line 92, in DownloadAndImport prog.Import(out=out) File "/home/mythbox/Scripts/mythnettv/program.py", line 472, in Import raise FilenameException(self.db, 'Could not determine the video ' program.FilenameException: Could not determine the video directory for this machine. Please report this to mythnettv@stillhq.com

The stack trace is mostly irrelevant. The problem here is that MythNetTV couldn’t decide what directory to put the video in once downloaded. Please execute the following SQL statements against your MythTV database:

    select * from storagegroup where groupname="MythNetTV";
    select * from storagegroup where groupname="Default";
    select * from settings where value="RecordFilePrefix";
    

This will dump all of the possible places MythNetTV will look for a video directory.

Once you’ve dumped this information, perform some simple checks:

  • Is there anything listed at all? If not, you need to configure storage groups with the MythTV user interface
  • Is there anything listed with the hostname that MythNetTV is running on? If not, you need to add configuration entries for your current hostname.

If you’re still having problems, please send email to mythnettv@stillhq.com, with the output of those select commands, and the output of the hostname command.

MythNetTV release 5

New things in this release:

  • There is now a users mailing list at
    http://lists.stillhq.com/listinfo.cgi/mythnettv-stillhq.com

  • Moved to a public SVN server at
    http://www.stillhq.com/mythtv/mythnettv/svn/

  • Added the ‘justone’ syntax to the download command
  • Another try at using gflags. This means that all the command lines have
    changed slightly.

  • Moved the core of MythTV out of the user interface file.
  • Started writing unit tests
  • Changed user output code so that it doesn’t insist on writing to stdout.
    You can now write to other file descriptors, which makes things like unit
    tests much easier to write.

  • Added video/msvideo to the enclosure whitelist
  • Added HTTP download progress information
  • Added a flag which turns off the prompts for markread (–noprompt)
  • Patches from Thomas Mashos
    • Search ~/.mythtv/mysql.txt, /usr/share/mythtv/mysql.txt and
      /etc/mythtv/mysql.txt in that order for MySQL connection information

    • A manpage
    • setup.py
  • video.py now has a simple command line interface to let you query it
  • Fix update of inactive programs bug per
    http://ubuntuforums.org/showpost.php?p=5580005&postcount=4

  • Better DB error handling
  • Included a COPYING file with the right version of the GPL (it was missing
    before)

  • Fixed a bug where programs would be downloaded more than once (found with
    a unit test!)

  • Started raising exceptions instead of just sys.exit(1). This should make
    life easier for user interfaces in the future

  • Default to using storage groups for storing recordings before falling back
    to the RecordFilePrefix. This makes the behaviour: use a storage group
    named “MythNetTV” if it exists; use the default storage group if it
    exists; use the value of RecordFilePrefix.

  • Transcode avc1 videos, because some need it
  • Force ASCII encoding of title, subtitle, and all fields in the database
    to get around feeds which use unicode which python / MySQL can’t store
    correctly

  • If there is only one attachment to an item, and its not in our whitelist
    of video formats, then warn the user that you’re assuming its a video file
    and then add it to the todo list

  • Slight tweak to the signature of video.MythNetTvVideo.Transcode()
  • Fix buf in RepairMissingDates which caused it to consistently crash
  • Fix typo in date warning code
  • Better handling of videos where the length of the video cannot be
    determined by mplayer

Release 5 is by far the best tested release of MythNetTV yet, with both unit tests and several users working quite closely with me to resolve problems found in the wild. You can grab your copy here.

Please help me test trunk

I’m preparing a new release of MythNetTV, and would like some help testing the code, as I have re-factored how the user interface works and there is some risk that I have broken things in the process. You can get the code like this:

    svn co http://www.stillhq.com/mythtv/mythnettv/svn
    

That will create a directory called mythnettv, with a subdirectory named trunk, which is the latest development version of the code. You should probably create that directory somewhere where you don’t mind a new directory being created.

The biggest change is that the command line syntax has changed slightly — the dashes have been removed from the commands. Therefore, to update your list of feeds, you now use:

      mythnettv update
    

Instead of:

      mythnettv --update
    

And so on. This was done so that I could add “real” flags, which are used to change default values like where the database configuration is read from, as well as what the default location for the temporary data directory is.

“Real” flags which are currently supported are:

      --datadirdefault:  The default location of the data directory
        (default: 'data')
    
      --db_host:  The name of the host the MySQL database is on,
                  don't define if you want to parse
                  ~/.mythtv/mysql.txt instead
        (default: '')
      --db_name:  The name of the database which MythNetTV uses,
                  don't define if you want to parse
                  ~/.mythtv/mysql.txt instead
        (default: '')
      --db_password:  The password for the database user, don't
                  define if you want to parse
                  ~/.mythtv/mysql.txt instead
        (default: '')
      --db_user:  The name of the user to connect to the database
                  with, don't define if you want to parse
                  ~/.mythtv/mysql.txt instead
        (default: '')
    
      --[no]commflag:  Run the mythcommflag command on new videos
        (default: 'true')
    

(These are the result of adding the gflags module back into the implementation).

I am hoping to release this version in the next few days, so if you find any bugs please send email to the mailing list.

MythNetTV release 4

New things in this release:

  • Removed gflags because people found it hard to find
  • Bug fix patch from David Linville applied
    • Fixed broken imports caused by refactoring
    • Transcode not needed for avc1 and m4v
  • Another bug fix patch from David Linville applied
    • Store filesize in the db
  • Removed some namespace imports I am not a fan of
  • Made verbosity optional for –update
  • Small code cleanups
  • Let the user know of repeated attempts to download a program
  • Documented the need for bittornado for bittorrent to work
  • Abandon downloading after 3 failed attempts (you can configure the number)
  • Detect stuck bittorrent downloads

Release 3 had a few bugs folks reported, this release should fix those problems. Let me know if you still see any. You can grab your copy here.

MythNetTV release 3

New things in this release:

  • Started work on an RSS exporter for MythTV recordings
  • DX50 doesn’t need transcode
  • Tweaked supported video mime types so “Tikibar TV” and “Ask a ninja” work
  • First cut of Bittorrent support
  • Schema upgrades
  • Archive recordings as well as importing them
  • Improved –list output
  • Subtitle restrictions on download as well
  • Make subscriptions inactive instead of deleting them (for unsubscribe)
  • Better filename safening
  • More markread options
  • Refactored code to be more sane
  • Don’t archive things imported from the archive location
  • Bulk import (–importmanylocal)

This version was a long time coming. Sorry about that. You can grab your copy here.