grahamc/hydra - Change JAH3UPWAVSHXIPNGL6PROQPZBYZHPJNFONWBDZX4HCX646USZXUQC

Support revision control systems via plugins

Created by Eelco Dolstra on May 25, 2013

JAH3UPWAVSHXIPNGL6PROQPZBYZHPJNFONWBDZX4HCX646USZXUQC

Dependencies

In channels

main

upstream

Change contents

Insertion in src/lib/Hydra/Controller/API.pm at line 221 [10.24]
[8.2376]
[9.418]
```
    # FIXME: injection danger.
```

Replacement in src/lib/Hydra/Controller/API.pm at line 226 [10.24]

∅:D[8.2402] → [9.462:520]

B:BD[9.462] → [9.462:520]

        my $clonePath = scmPath . "/" . sha256_hex($uri);

[8.2402]

[9.520]

        my $clonePath = getSCMCacheDir . "/hg/" . sha256_hex($uri);

Replacement in src/lib/Hydra/Controller/API.pm at line 232 [10.24]

B:BD[9.690] → [11.0:58]

        my $clonePath = scmPath . "/" . sha256_hex($uri);

[9.690]

[9.756]

        my $clonePath = getSCMCacheDir . "/git/" . sha256_hex($uri);

Replacement in src/lib/Hydra/Controller/Build.pm at line 536 [14.1]

∅:D[12.395] → [13.680:746]

B:BD[13.680] → [13.680:746]

                $c->model('DB'), $build->project, $build->jobset,

[12.395]

[13.746]

                $c->hydra_plugins, $c->model('DB'), $build->project, $build->jobset,

Insertion in src/lib/Hydra/Controller/Root.pm at line 31 [16.804]

[15.374]

[16.1351]


    # Gather the supported input types.
    $c->stash->{inputTypes} = {
        'string' => 'String value',
        'boolean' => 'Boolean',
        'build' => 'Build output',
        'sysbuild' => 'Build output (same system)'
    };
    $_->supportedInputTypes($c->stash->{inputTypes}) foreach @{$c->hydra_plugins};

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 6 [13.2663]
∅:D[12.1289] → [13.2711:2735]
B:BD[13.2711] → [13.2711:2735]
```
use POSIX qw(strftime);
```

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 21 [13.2663]

B:BD[17.18] → [18.0:68]

B:BD[18.68] → [19.787:844]

    fetchInput evalJobs checkBuild inputsToArgs captureStdoutStderr
    getReleaseName addBuildProducts restartBuild scmPath

[17.18]

[20.67]

    fetchInput evalJobs checkBuild inputsToArgs
    getReleaseName addBuildProducts restartBuild

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 25 [13.2663]
∅:D[21.63] → [13.2815:2816]
∅:D[22.98] → [13.2815:2816]
∅:D[23.110] → [13.2815:2816]
∅:D[24.127] → [13.2815:2816]
∅:D[17.159] → [13.2815:2816]
∅:D[25.307] → [13.2815:2816]
∅:D[26.774] → [13.2815:2816]
∅:D[9.1152] → [13.2815:2816]
∅:D[12.1355] → [13.2815:2816]
B:BD[13.2815] → [13.2815:2816]
B:BD[13.2816] → [17.160:161]
∅:D[17.161] → [27.32:46]
B:BD[13.2816] → [27.32:46]
B:BD[27.46] → [28.23:76]
∅:D[28.76] → [23.111:115]
B:BD[27.81] → [23.111:115]
∅:D[23.266] → [13.2817:3151]
B:BD[13.2817] → [13.2817:3151]
B:BD[13.3151] → [18.191:209]
∅:D[18.209] → [13.3173:3175]
B:BD[13.3173] → [13.3173:3175]
```
sub scmPath {
    return Hydra::Model::DB::getHydraPath . "/scm" ;
}
sub getStorePathHash {
    my ($storePath) = @_;
    my $hash = `nix-store --query --hash $storePath`
        or die "cannot get hash of $storePath";
    chomp $hash;
    die unless $hash =~ /^sha256:(.*)$/;
    $hash = $1;
    $hash = `nix-hash --to-base16 --type sha256 $hash`
        or die "cannot convert hash";
    chomp $hash;
    return $hash;
}
```

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 72 [13.2663]

B:BD[13.4413] → [13.4413:4415]

B:BD[13.4415] → [29.0:1]

∅:D[29.1] → [13.4416:4417]

B:BD[13.4416] → [13.4416:4417]

B:BD[13.4417] → [30.156:177]

B:BD[30.177] → [31.0:53]

∅:D[31.53] → [13.4494:4495]

B:BD[13.4494] → [13.4494:4495]

B:BD[13.4495] → [29.2:24]

∅:D[29.24] → [13.4548:4549]

B:BD[13.4548] → [13.4548:4549]

B:BD[13.4549] → [29.25:86]

∅:D[29.86] → [13.4622:4623]

B:BD[13.4622] → [13.4622:4623]

B:BD[13.4623] → [29.87:347]

∅:D[29.347] → [13.4899:4900]

B:BD[13.4899] → [13.4899:4900]

B:BD[13.4900] → [29.348:569]

∅:D[29.569] → [13.5141:5142]

B:BD[13.5141] → [13.5141:5142]

B:BD[13.5142] → [29.570:678]

B:BD[29.678] → [4.76:140]

∅:D[4.140] → [29.742:768]

B:BD[29.742] → [29.742:768]

∅:D[29.768] → [13.5356:5357]

B:BD[13.5356] → [13.5356:5357]

B:BD[13.5357] → [29.769:816]

∅:D[29.816] → [13.5408:5409]

B:BD[13.5408] → [13.5408:5409]

B:BD[13.5409] → [29.817:941]

∅:D[29.941] → [13.5541:5542]

B:BD[13.5541] → [13.5541:5542]

B:BD[13.5542] → [29.942:1350]

B:BD[29.1350] → [3.0:70]

∅:D[3.70] → [29.1410:1653]

B:BD[29.1410] → [29.1410:1653]

∅:D[29.1653] → [13.6323:6343]

B:BD[13.6323] → [13.6323:6343]

B:BD[13.6343] → [29.1654:1825]

∅:D[29.1825] → [13.6548:6558]

B:BD[13.6548] → [13.6548:6558]

B:BD[13.6558] → [29.1826:1832]

∅:D[29.1832] → [13.6558:6559]

B:BD[13.6558] → [13.6558:6559]

B:BD[13.6559] → [29.1833:1844]

B:BD[29.1844] → [31.54:76]

∅:D[31.76] → [29.1890:2033]

B:BD[29.1890] → [29.1890:2033]

∅:D[29.2033] → [30.178:181]

B:BD[13.6787] → [30.178:181]

B:BD[30.181] → [29.2034:2035]

∅:D[29.2035] → [30.181:201]

B:BD[30.181] → [30.181:201]

B:BD[30.201] → [31.77:141]

∅:D[31.141] → [13.6793:6794]

∅:D[32.185] → [13.6793:6794]

∅:D[30.261] → [13.6793:6794]

B:BD[13.6793] → [13.6793:6794]

B:BD[13.6794] → [33.0:110]

∅:D[33.110] → [13.6849:6850]

∅:D[29.2058] → [13.6849:6850]

B:BD[13.6849] → [13.6849:6850]

B:BD[13.6850] → [29.2059:2094]

B:BD[29.2274] → [29.2274:2302]

B:BD[29.2302] → [33.111:145]

B:BD[33.145] → [34.26:217]

B:BD[34.217] → [4.141:281]

∅:D[4.281] → [35.0:66]

B:BD[34.361] → [35.0:66]

∅:D[35.66] → [33.477:484]

∅:D[34.407] → [33.477:484]

B:BD[33.477] → [33.477:484]

∅:D[33.484] → [29.2483:2520]

B:BD[29.2483] → [29.2483:2520]

∅:D[29.2520] → [13.7351:7352]

B:BD[13.7351] → [13.7351:7352]

B:BD[13.7352] → [36.17:190]

∅:D[36.190] → [29.2521:2642]

B:BD[13.7352] → [29.2521:2642]

∅:D[29.2642] → [13.7481:7482]

B:BD[13.7481] → [13.7481:7482]

B:BD[13.7482] → [29.2643:2818]

B:BD[29.2818] → [36.191:192]

B:BD[36.192] → [34.408:515]

B:BD[34.515] → [2.0:77]

∅:D[2.77] → [36.381:493]

∅:D[34.625] → [36.381:493]

B:BD[36.381] → [36.381:493]

B:BD[36.493] → [4.282:475]

∅:D[4.475] → [36.560:561]

B:BD[36.560] → [36.560:561]

B:BD[36.561] → [34.626:1162]

∅:D[36.1034] → [13.8133:8134]

∅:D[34.1162] → [13.8133:8134]

∅:D[29.3238] → [13.8133:8134]

B:BD[13.8133] → [13.8133:8134]

B:BD[13.8134] → [36.1035:1104]

∅:D[36.1104] → [13.8190:8191]

∅:D[29.3291] → [13.8190:8191]

B:BD[13.8190] → [13.8190:8191]

B:BD[13.8191] → [29.3292:3318]

B:BD[29.3318] → [36.1105:1177]

∅:D[36.1177] → [29.3380:3552]

B:BD[29.3380] → [29.3380:3552]

∅:D[29.3552] → [13.8479:8495]

B:BD[13.8479] → [13.8479:8495]

B:BD[13.8495] → [29.3553:3559]

∅:D[29.3559] → [13.8505:8506]

B:BD[13.8505] → [13.8505:8506]

B:BD[13.8506] → [18.210:221]

∅:D[18.221] → [31.142:164]

B:BD[29.3572] → [31.142:164]

∅:D[31.164] → [29.3618:3727]

B:BD[29.3618] → [29.3618:3727]

}
sub fetchInputPath {
    my ($db, $project, $jobset, $name, $value) = @_;
    my $uri = $value;
    my $timestamp = time;
    my $sha256;
    my $storePath;
    # Some simple caching: don't check a path more than once every N seconds.
    (my $cachedInput) = $db->resultset('CachedPathInputs')->search(
        {srcpath => $uri, lastseen => {">", $timestamp - 30}},
        {rows => 1, order_by => "lastseen DESC"});
    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
        $storePath = $cachedInput->storepath;
        $sha256 = $cachedInput->sha256hash;
        $timestamp = $cachedInput->timestamp;
    } else {
        print STDERR "copying input ", $name, " from $uri\n";
        $storePath = `nix-store --add "$uri"`
            or die "cannot copy path $uri to the Nix store.\n";
        chomp $storePath;
        $sha256 = getStorePathHash $storePath;
        ($cachedInput) = $db->resultset('CachedPathInputs')->search(
            {srcpath => $uri, sha256hash => $sha256});
        # Path inputs don't have a natural notion of a "revision", so
        # we simulate it by using the timestamp that we first saw this
        # path have this SHA-256 hash.  So if the contents of the path
        # changes, we get a new "revision", but if it doesn't change
        # (or changes back), we don't get a new "revision".
        if (!defined $cachedInput) {
            txn_do($db, sub {
                $db->resultset('CachedPathInputs')->update_or_create(
                    { srcpath => $uri
                    , timestamp => $timestamp
                    , lastseen => $timestamp
                    , sha256hash => $sha256
                    , storepath => $storePath
                    });
                });
        } else {
            $timestamp = $cachedInput->timestamp;
            txn_do($db, sub {
                $cachedInput->update({lastseen => time});
            });
        }
    }
    return
        { uri => $uri
        , storePath => $storePath
        , sha256hash => $sha256
        , revision => strftime "%Y%m%d%H%M%S", gmtime($timestamp)
        };
}
sub fetchInputSVN {
    my ($db, $project, $jobset, $name, $value, $checkout) = @_;
    # Allow users to specify a revision number next to the URI.
    my ($uri, $revision) = split ' ', $value;
    my $sha256;
    my $storePath;
    my $stdout; my $stderr;
    unless (defined $revision) {
        # First figure out the last-modified revision of the URI.
        my @cmd = (["svn", "ls", "-v", "--depth", "empty", $uri],
                   "|", ["sed", 's/^ *\([0-9]*\).*/\1/']);
        IPC::Run::run(@cmd, \$stdout, \$stderr);
        die "cannot get head revision of Subversion repository at `$uri':\n$stderr" if $?;
        $revision = $stdout; $revision =~ s/\s*([0-9]+)\s*/$1/sm;
    }
    die unless $revision =~ /^\d+$/;
    # Do we already have this revision in the store?
    # !!! This needs to take $checkout into account!  Otherwise "svn"
    # and "svn-checkout" inputs can get mixed up.
    (my $cachedInput) = $db->resultset('CachedSubversionInputs')->search(
        {uri => $uri, revision => $revision});
    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
        $storePath = $cachedInput->storepath;
        $sha256 = $cachedInput->sha256hash;
    } else {
        # No, do a checkout.  The working copy is reused between
        # invocations to speed things up.
        my $wcPath = scmPath . "/svn/" . sha256_hex($uri) . "/svn-checkout";
        print STDERR "checking out Subversion input ", $name, " from $uri revision $revision into $wcPath\n";
        (my $res, $stdout, $stderr) = captureStdoutStderr(600, "svn", "checkout", $uri, "-r", $revision, $wcPath);
        die "error checking out Subversion repo at `$uri':\n$stderr" if $res;
        if ($checkout) {
            $storePath = addToStore($wcPath, 1, "sha256");
        } else {
            # Hm, if the Nix Perl bindings supported filters in
            # addToStore(), then we wouldn't need to make a copy here.
            my $tmpDir = File::Temp->newdir("hydra-svn-export.XXXXXX", CLEANUP => 1, TMPDIR => 1) or die;
            (system "svn", "export", $wcPath, "$tmpDir/svn-export", "--quiet") == 0 or die "svn export failed";
            $storePath = addToStore("$tmpDir/svn-export", 1, "sha256");
        }
        $sha256 = queryPathHash($storePath); $sha256 =~ s/sha256://;
        txn_do($db, sub {
            $db->resultset('CachedSubversionInputs')->update_or_create(
                { uri => $uri
                , revision => $revision
                , sha256hash => $sha256
                , storepath => $storePath
                });
            });
    }
    return
        { uri => $uri
        , storePath => $storePath
        , sha256hash => $sha256
        , revision => $revision
        };

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 127 [13.2663]
B:BD[29.5444] → [29.5444:5466]
```
        return undef;
```
[29.5444]
[29.5466]
```
        return ();
```

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 150 [13.2663]

∅:D[29.6139] → [37.870:871]

B:BD[37.870] → [37.870:871]

B:BD[30.350] → [30.350:371]

B:BD[30.371] → [31.365:418]

∅:D[31.418] → [30.431:432]

B:BD[30.431] → [30.431:432]

B:BD[30.432] → [38.0:114]

∅:D[38.114] → [39.100:101]

B:BD[39.100] → [39.100:101]

∅:D[39.101] → [30.454:515]

B:BD[30.454] → [30.454:515]

B:BD[30.515] → [40.0:1]

B:BD[27.103] → [27.103:124]

B:BD[27.124] → [41.0:54]

∅:D[41.54] → [27.179:180]

∅:D[42.61] → [27.179:180]

∅:D[43.62] → [27.179:180]

∅:D[11.110] → [27.179:180]

∅:D[36.1232] → [27.179:180]

B:BD[27.179] → [27.179:180]

B:BD[27.180] → [38.115:162]

∅:D[38.162] → [27.181:208]

B:BD[40.120] → [27.181:208]

B:BD[27.208] → [38.163:318]

B:BD[38.318] → [4.476:658]

∅:D[4.658] → [27.390:397]

B:BD[27.390] → [27.390:397]

B:BD[27.424] → [17.162:227]

B:BD[17.227] → [38.379:380]

B:BD[38.380] → [44.0:264]

B:BD[44.264] → [4.659:721]

∅:D[4.721] → [45.0:56]

B:BD[44.328] → [45.0:56]

B:BD[45.56] → [4.722:854]

∅:D[44.414] → [27.563:564]

∅:D[4.854] → [27.563:564]

∅:D[38.1289] → [27.563:564]

B:BD[27.563] → [27.563:564]

B:BD[27.564] → [38.1290:1388]

B:BD[38.1388] → [4.855:948]

∅:D[46.151] → [30.515:516]

∅:D[40.308] → [30.515:516]

∅:D[27.783] → [30.515:516]

∅:D[4.948] → [30.515:516]

∅:D[38.1485] → [30.515:516]

B:BD[30.515] → [30.515:516]

B:BD[30.516] → [38.1486:1528]

B:BD[38.1528] → [4.949:1095]

∅:D[4.1095] → [40.405:406]

∅:D[38.1642] → [40.405:406]

B:BD[40.405] → [40.405:406]

B:BD[40.406] → [38.1643:1890]

B:BD[38.1890] → [4.1096:1270]

∅:D[4.1270] → [47.237:238]

∅:D[34.1403] → [47.237:238]

∅:D[38.2082] → [47.237:238]

B:BD[47.237] → [47.237:238]

B:BD[47.238] → [38.2083:2308]

B:BD[38.2308] → [4.1271:1420]

∅:D[4.1420] → [38.2464:2474]

B:BD[38.2464] → [38.2464:2474]

∅:D[34.1496] → [47.323:330]

∅:D[38.2474] → [47.323:330]

B:BD[47.323] → [47.323:330]

B:BD[47.330] → [38.2475:2621]

∅:D[38.2621] → [40.532:618]

B:BD[40.532] → [40.532:618]

B:BD[40.618] → [29.6185:6272]

∅:D[40.918] → [30.759:832]

B:BD[30.759] → [30.759:832]

B:BD[30.832] → [48.0:46]

B:BD[48.46] → [29.6481:6525]

B:BD[29.6571] → [29.6571:6615]

∅:D[48.171] → [30.947:960]

∅:D[40.956] → [30.947:960]

∅:D[29.6615] → [30.947:960]

B:BD[30.947] → [30.947:960]

B:BD[48.177] → [29.6616:6670]

B:BD[29.6670] → [38.2622:2690]

∅:D[49.59] → [29.6727:6803]

∅:D[38.2690] → [29.6727:6803]

B:BD[29.6727] → [29.6727:6803]

B:BD[29.6803] → [38.2691:3441]


sub fetchInputGit {
    my ($db, $project, $jobset, $name, $value) = @_;
    (my $uri, my $branch, my $deepClone) = split ' ', $value;
    $branch = defined $branch ? $branch : "master";
    my $timestamp = time;
    my $sha256;
    my $storePath;
    mkpath(scmPath);
    my $clonePath = scmPath . "/" . sha256_hex($uri);
    my $stdout = ""; my $stderr = ""; my $res;
    if (! -d $clonePath) {
        # Clone everything and fetch the branch.
        # TODO: Optimize the first clone by using "git init $clonePath" and "git remote add origin $uri".
        ($res, $stdout, $stderr) = captureStdoutStderr(600, "git", "clone", "--branch", $branch, $uri, $clonePath);
        die "error cloning git repo at `$uri':\n$stderr" if $res;
    }
    chdir $clonePath or die $!; # !!! urgh, shouldn't do a chdir
    # This command force the update of the local branch to be in the same as
    # the remote branch for whatever the repository state is.  This command mirror
    # only one branch of the remote repository.
    ($res, $stdout, $stderr) = captureStdoutStderr(600,
        "git", "fetch", "-fu", "origin", "+$branch:$branch");
    ($res, $stdout, $stderr) = captureStdoutStderr(600,
        "git", "fetch", "-fu", "origin") if $res;
    die "error fetching latest change from git repo at `$uri':\n$stderr" if $res;
    ($res, $stdout, $stderr) = captureStdoutStderr(600,
        ("git", "rev-parse", "$branch"));
    die "error getting revision number of Git branch '$branch' at `$uri':\n$stderr" if $res;
    my ($revision) = split /\n/, $stdout;
    die "error getting a well-formated revision number of Git branch '$branch' at `$uri':\n$stdout"
        unless $revision =~ /^[0-9a-fA-F]+$/;
    my $ref = "refs/heads/$branch";
    # If deepClone is defined, then we look at the content of the repository
    # to determine if this is a top-git branch.
    if (defined $deepClone) {
        # Checkout the branch to look at its content.
        ($res, $stdout, $stderr) = captureStdoutStderr(600, "git", "checkout", "$branch");
        die "error checking out Git branch '$branch' at `$uri':\n$stderr" if $res;
        if (-f ".topdeps") {
            # This is a TopGit branch.  Fetch all the topic branches so
            # that builders can run "tg patch" and similar.
            ($res, $stdout, $stderr) = captureStdoutStderr(600,
                "tg", "remote", "--populate", "origin");
            print STDERR "warning: `tg remote --populate origin' failed:\n$stderr" if $res;
        }
    }
    # Some simple caching: don't check a uri/branch/revision more than once.
    # TODO: Fix case where the branch is reset to a previous commit.
    my $cachedInput ;
    ($cachedInput) = $db->resultset('CachedGitInputs')->search(
        {uri => $uri, branch => $branch, revision => $revision},
        {rows => 1});
    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
        $storePath = $cachedInput->storepath;
        $sha256 = $cachedInput->sha256hash;
        $revision = $cachedInput->revision;
    } else {
        # Then download this revision into the store.
        print STDERR "checking out Git branch $branch from $uri\n";
        $ENV{"NIX_HASH_ALGO"} = "sha256";
        $ENV{"PRINT_PATH"} = "1";
        $ENV{"NIX_PREFETCH_GIT_LEAVE_DOT_GIT"} = "0";
        $ENV{"NIX_PREFETCH_GIT_DEEP_CLONE"} = "";
        if (defined $deepClone) {
            # Checked out code often wants to be able to run `git
            # describe', e.g., code that uses Gnulib's `git-version-gen'
            # script.  Thus, we leave `.git' in there.  Same for
            # Subversion (e.g., libgcrypt's build system uses that.)
            $ENV{"NIX_PREFETCH_GIT_LEAVE_DOT_GIT"} = "1";
            # Ask for a "deep clone" to allow "git describe" and similar
            # tools to work.  See
            # http://thread.gmane.org/gmane.linux.distributions.nixos/3569
            # for a discussion.
            $ENV{"NIX_PREFETCH_GIT_DEEP_CLONE"} = "1";
        }

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 151 [13.2663]

B:BD[50.1] → [4.1421:1617]

        ($res, $stdout, $stderr) = captureStdoutStderr(600, "nix-prefetch-git", $clonePath, $revision);
        die "cannot check out Git repository branch '$branch' at `$uri':\n$stderr" if $res;

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 152 [13.2663]

∅:D[50.3] → [29.7626:7678]

B:BD[48.1149] → [29.7626:7678]

B:BD[29.7678] → [18.267:268]

∅:D[18.268] → [27.784:1063]

B:BD[48.1335] → [27.784:1063]

∅:D[27.1063] → [29.8276:8296]

B:BD[29.8276] → [29.8276:8296]

B:BD[48.1991] → [29.8394:8410]

∅:D[30.1921] → [13.10008:10014]

∅:D[48.2011] → [13.10008:10014]

∅:D[29.8420] → [13.10008:10014]

B:BD[13.10008] → [13.10008:10014]

B:BD[13.10014] → [50.4:73]

B:BD[50.73] → [51.0:175]

∅:D[51.175] → [50.199:420]

B:BD[50.199] → [50.199:420]

B:BD[50.420] → [51.176:290]

∅:D[51.290] → [30.1922:1934]

∅:D[50.420] → [30.1922:1934]

B:BD[13.10014] → [30.1922:1934]

B:BD[30.1934] → [31.419:441]

∅:D[31.441] → [29.8467:8565]

B:BD[29.8467] → [29.8467:8565]

B:BD[29.8565] → [50.421:486]

B:BD[50.486] → [51.291:323]

∅:D[51.323] → [29.8565:8578]

∅:D[50.486] → [29.8565:8578]

B:BD[29.8565] → [29.8565:8578]

B:BD[29.8578] → [52.92:93]

B:BD[52.93] → [17.228:229]

∅:D[17.229] → [52.93:116]

B:BD[52.93] → [52.93:116]

B:BD[52.116] → [31.442:506]

∅:D[31.506] → [52.187:275]

B:BD[52.187] → [52.187:275]

B:BD[52.275] → [36.1233:1234]

∅:D[36.1234] → [52.294:315]

B:BD[52.294] → [52.294:315]

B:BD[52.315] → [36.1235:1289]

∅:D[36.1289] → [52.366:394]

B:BD[52.366] → [52.366:394]

B:BD[52.394] → [4.1618:1788]

∅:D[4.1788] → [52.582:621]

B:BD[52.582] → [52.582:621]

B:BD[52.621] → [4.1789:1945]

∅:D[4.1945] → [52.791:854]

B:BD[52.791] → [52.791:854]

B:BD[52.854] → [4.1946:2019]

∅:D[4.2019] → [52.942:943]

B:BD[52.942] → [52.942:943]

B:BD[52.943] → [4.2020:2144]

∅:D[4.2144] → [52.1071:1447]

B:BD[52.1071] → [52.1071:1447]

B:BD[52.1447] → [18.269:270]

∅:D[18.270] → [52.1460:1745]

B:BD[52.1460] → [52.1460:1745]

B:BD[52.1745] → [18.271:272]

∅:D[18.272] → [52.1754:1817]

B:BD[52.1754] → [52.1754:1817]

B:BD[52.1817] → [4.2145:2272]

∅:D[4.2272] → [52.1950:2282]

B:BD[52.1950] → [52.1950:2282]

∅:D[52.2282] → [30.2117:2118]

∅:D[29.8578] → [30.2117:2118]

B:BD[30.2117] → [30.2117:2118]

B:BD[30.2118] → [18.273:284]

∅:D[18.284] → [31.507:529]

B:BD[52.2295] → [31.507:529]

∅:D[31.529] → [52.2341:2439]

B:BD[52.2341] → [52.2341:2439]

B:BD[52.2439] → [53.4:18]

B:BD[53.18] → [18.285:286]

∅:D[18.286] → [54.249:268]

∅:D[52.2456] → [54.249:268]

B:BD[30.2121] → [54.249:268]

B:BD[54.268] → [31.530:583]

B:BD[31.583] → [18.287:288]

∅:D[18.288] → [55.0:43]

B:BD[54.274] → [55.0:43]

B:BD[55.43] → [56.0:41]

∅:D[56.41] → [54.374:402]

∅:D[55.43] → [54.374:402]

B:BD[54.374] → [54.374:402]

B:BD[54.402] → [38.3503:3541]

∅:D[38.3541] → [36.1290:1291]

B:BD[54.430] → [36.1290:1291]

∅:D[36.1291] → [54.449:470]

B:BD[54.449] → [54.449:470]

B:BD[54.470] → [36.1292:1346]

∅:D[36.1346] → [54.521:612]

B:BD[54.521] → [54.521:612]

B:BD[54.612] → [4.2273:2391]

∅:D[4.2391] → [54.736:801]

B:BD[54.736] → [54.736:801]

B:BD[54.801] → [4.2392:2548]

∅:D[4.2548] → [54.971:972]

B:BD[54.971] → [54.971:972]

B:BD[54.972] → [4.2549:2770]

∅:D[55.258] → [54.1142:1143]

∅:D[27.1140] → [54.1142:1143]

∅:D[4.2770] → [54.1142:1143]

B:BD[54.1142] → [54.1142:1143]

B:BD[54.1143] → [57.132:193]

B:BD[57.193] → [18.289:290]

∅:D[18.290] → [54.1295:1806]

B:BD[54.1295] → [54.1295:1806]

B:BD[54.1806] → [18.291:292]

∅:D[18.292] → [54.1815:1878]

B:BD[54.1815] → [54.1815:1878]

B:BD[54.1878] → [4.2771:2904]

∅:D[4.2904] → [54.2011:2091]

B:BD[54.2011] → [54.2011:2091]

B:BD[54.2091] → [3.71:135]

∅:D[3.135] → [54.2145:2376]

B:BD[54.2145] → [54.2145:2376]

B:BD[54.2376] → [18.293:304]

∅:D[18.304] → [31.584:606]

B:BD[54.2388] → [31.584:606]

∅:D[31.606] → [54.2434:2560]

B:BD[54.2434] → [54.2434:2560]

B:BD[54.2560] → [57.194:231]

∅:D[57.231] → [54.2560:2571]

B:BD[54.2560] → [54.2560:2571]

∅:D[54.2576] → [30.2201:2203]

B:BD[30.2201] → [30.2201:2203]

B:BD[30.2203] → [29.8579:8580]

∅:D[29.8580] → [30.2203:2204]

B:BD[30.2203] → [30.2203:2204]

        ($sha256, $storePath) = split ' ', $stdout;
        txn_do($db, sub {
            $db->resultset('CachedGitInputs')->update_or_create(
                { uri => $uri
                , branch => $branch
                , revision => $revision
                , sha256hash => $sha256
                , storepath => $storePath
                });
            });
    }
    # For convenience in producing readable version names, pass the
    # number of commits in the history of this revision (‘revCount’)
    # the output of git-describe (‘gitTag’), and the abbreviated
    # revision (‘shortRev’).
    my $revCount = `git rev-list $revision | wc -l`; chomp $revCount;
    die "git rev-list failed" if $? != 0;
    my $gitTag = `git describe --always $revision`; chomp $gitTag;
    die "git describe failed" if $? != 0;
    my $shortRev = `git rev-parse --short $revision`; chomp $shortRev;
    die "git rev-parse failed" if $? != 0;
    return
        { uri => $uri
        , storePath => $storePath
        , sha256hash => $sha256
        , revision => $revision
        , revCount => int($revCount)
        , gitTag => $gitTag
        , shortRev => $shortRev
        };
}
sub fetchInputBazaar {
    my ($db, $project, $jobset, $name, $value, $checkout) = @_;
    my $uri = $value;
    my $sha256;
    my $storePath;
    my $stdout; my $stderr;
    mkpath(scmPath);
    my $clonePath = scmPath . "/" . sha256_hex($uri);
    if (! -d $clonePath) {
        (my $res, $stdout, $stderr) = captureStdoutStderr(600, "bzr", "branch", $uri, $clonePath);
        die "error cloning bazaar branch at `$uri':\n$stderr" if $res;
    }
    chdir $clonePath or die $!;
    (my $res, $stdout, $stderr) = captureStdoutStderr(600, "bzr", "pull");
    die "error pulling latest change bazaar branch at `$uri':\n$stderr" if $res;
    # First figure out the last-modified revision of the URI.
    my @cmd = (["bzr", "revno"], "|", ["sed", 's/^ *\([0-9]*\).*/\1/']);
    IPC::Run::run(@cmd, \$stdout, \$stderr);
    die "cannot get head revision of Bazaar branch at `$uri':\n$stderr" if $?;
    my $revision = $stdout; chomp $revision;
    die unless $revision =~ /^\d+$/;
    (my $cachedInput) = $db->resultset('CachedBazaarInputs')->search(
        {uri => $uri, revision => $revision});
    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
        $storePath = $cachedInput->storepath;
        $sha256 = $cachedInput->sha256hash;
    } else {
        # Then download this revision into the store.
        print STDERR "checking out Bazaar input ", $name, " from $uri revision $revision\n";
        $ENV{"NIX_HASH_ALGO"} = "sha256";
        $ENV{"PRINT_PATH"} = "1";
        $ENV{"NIX_PREFETCH_BZR_LEAVE_DOT_BZR"} = "$checkout";
        (my $res, $stdout, $stderr) = captureStdoutStderr(600,
            "nix-prefetch-bzr", $clonePath, $revision);
        die "cannot check out Bazaar branch `$uri':\n$stderr" if $res;
        ($sha256, $storePath) = split ' ', $stdout;
        txn_do($db, sub {
            $db->resultset('CachedBazaarInputs')->create(
                { uri => $uri
                , revision => $revision
                , sha256hash => $sha256
                , storepath => $storePath
                });
            });
    }
    return
        { uri => $uri
        , storePath => $storePath
        , sha256hash => $sha256
        , revision => $revision
        };
}
sub fetchInputHg {
    my ($db, $project, $jobset, $name, $value) = @_;
    (my $uri, my $id) = split ' ', $value;
    $id = defined $id ? $id : "default";
    # init local hg clone
    my $stdout = ""; my $stderr = "";
    mkpath(scmPath);
    my $clonePath = scmPath . "/" . sha256_hex($uri);
    if (! -d $clonePath) {
        (my $res, $stdout, $stderr) = captureStdoutStderr(600,
            "hg", "clone", $uri, $clonePath);
        die "error cloning mercurial repo at `$uri':\n$stderr" if $res;
    }
    # hg pull + check rev
    chdir $clonePath or die $!;
    (my $res, $stdout, $stderr) = captureStdoutStderr(600, "hg", "pull");
    die "error pulling latest change mercurial repo at `$uri':\n$stderr" if $res;
    (my $res1, $stdout, $stderr) = captureStdoutStderr(600,
        "hg", "log", "-r", $id, "--template", "{node|short} {rev} {branch}");
    die "error getting branch and revision of $id from `$uri':\n$stderr" if $res1;
    my ($revision, $revCount, $branch) = split ' ', $stdout;
    my $storePath;
    my $sha256;
    (my $cachedInput) = $db->resultset('CachedHgInputs')->search(
        {uri => $uri, branch => $branch, revision => $revision});
    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
        $storePath = $cachedInput->storepath;
        $sha256 = $cachedInput->sha256hash;
    } else {
        print STDERR "checking out Mercurial input from $uri $branch revision $revision\n";
        $ENV{"NIX_HASH_ALGO"} = "sha256";
        $ENV{"PRINT_PATH"} = "1";
        (my $res, $stdout, $stderr) = captureStdoutStderr(600,
            "nix-prefetch-hg", $clonePath, $revision);
        die "cannot check out Mercurial repository `$uri':\n$stderr" if $res;
        ($sha256, $storePath) = split ' ', $stdout;
        txn_do($db, sub {
            $db->resultset('CachedHgInputs')->update_or_create(
                { uri => $uri
                , branch => $branch
                , revision => $revision
                , sha256hash => $sha256
                , storepath => $storePath
                });
            });
    }
    return
        { uri => $uri
        , branch => $branch
        , storePath => $storePath
        , sha256hash => $sha256
        , revision => $revision
        , revCount => int($revCount)
        };
}

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 153 [13.2663]

B:BD[30.2221] → [30.2221:2281]

    my ($db, $project, $jobset, $name, $type, $value) = @_;

[30.2221]

[58.0]

    my ($plugins, $db, $project, $jobset, $name, $type, $value) = @_;

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 156 [13.2663]

B:BD[30.2282] → [30.2282:2309]

B:BD[30.2309] → [58.17:94]

∅:D[58.94] → [30.2379:2414]

∅:D[31.695] → [30.2379:2414]

∅:D[29.8658] → [30.2379:2414]

B:BD[30.2379] → [30.2379:2414]

B:BD[30.2414] → [58.95:174]

∅:D[58.174] → [32.337:381]

∅:D[31.770] → [32.337:381]

B:BD[32.337] → [32.337:381]

B:BD[32.381] → [58.175:254]

∅:D[58.254] → [30.2483:2520]

∅:D[32.460] → [30.2483:2520]

∅:D[31.845] → [30.2483:2520]

∅:D[29.8735] → [30.2483:2520]

B:BD[30.2483] → [30.2483:2520]

B:BD[30.2520] → [58.255:333]

    if ($type eq "path") {
        push @inputs, fetchInputPath($db, $project, $jobset, $name, $value);
    }
    elsif ($type eq "svn") {
        push @inputs, fetchInputSVN($db, $project, $jobset, $name, $value, 0);
    }
    elsif ($type eq "svn-checkout") {
        push @inputs, fetchInputSVN($db, $project, $jobset, $name, $value, 1);
    }
    elsif ($type eq "build") {
        push @inputs, fetchInputBuild($db, $project, $jobset, $name, $value);

[30.2282]

[37.1515]

    if ($type eq "build") {
        @inputs = fetchInputBuild($db, $project, $jobset, $name, $value);

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 160 [13.2663]

B:BD[37.1555] → [58.334:418]

        push @inputs, fetchInputSystemBuild($db, $project, $jobset, $name, $value);

[37.1555]

[30.2591]

        @inputs = fetchInputSystemBuild($db, $project, $jobset, $name, $value);

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 162 [13.2663]
B:BD[30.2597] → [30.2597:2626]
B:BD[30.2626] → [58.419:495]
∅:D[58.495] → [30.2695:2701]
∅:D[31.1071] → [30.2695:2701]
∅:D[29.8976] → [30.2695:2701]
B:BD[30.2695] → [30.2695:2701]
B:BD[30.2701] → [59.0:28]
B:BD[59.28] → [58.496:571]
∅:D[58.571] → [59.103:109]
∅:D[31.1142] → [59.103:109]
B:BD[59.103] → [59.103:109]
B:BD[59.109] → [52.2457:2486]
B:BD[52.2486] → [58.572:654]
∅:D[58.654] → [52.2568:2612]
∅:D[31.1220] → [52.2568:2612]
B:BD[52.2568] → [52.2568:2612]
B:BD[52.2612] → [58.655:737]
B:BD[58.737] → [18.305:311]
```
    elsif ($type eq "git") {
        push @inputs, fetchInputGit($db, $project, $jobset, $name, $value);
    }
    elsif ($type eq "hg") {
        push @inputs, fetchInputHg($db, $project, $jobset, $name, $value);
    }
    elsif ($type eq "bzr") {
        push @inputs, fetchInputBazaar($db, $project, $jobset, $name, $value, 0);
    }
    elsif ($type eq "bzr-checkout") {
        push @inputs, fetchInputBazaar($db, $project, $jobset, $name, $value, 1);
    }
```
Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 164 [13.2663]
B:BD[13.10086] → [58.738:781]
```
        push @inputs, { value => $value };
```
[13.10086]
[18.312]
```
        @inputs = { value => $value };
```
Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 168 [13.2663]
B:BD[13.10257] → [58.782:825]
```
        push @inputs, { value => $value };
```
[13.10257]
[18.319]
```
        @inputs = { value => $value };
```

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 171 [13.2663]

B:BD[13.10328] → [4.2905:2968]

        die "input `" . $name . "' has unknown type `$type'.";

[13.10328]

[13.10391]

        my $found = 0;
        foreach my $plugin (@{$plugins}) {
            @inputs = $plugin->fetchInput($type, $name, $value);
            if (defined $inputs[0]) {
                $found = 1;
                last;
            }
        }
        die "input `$name' has unknown type `$type'." unless $found;

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 182 [13.2663]
B:BD[31.1378] → [18.326:360]
B:BD[18.360] → [60.2781:2831]
∅:D[60.2831] → [58.904:910]
B:BD[58.904] → [58.904:910]
```
    foreach my $input (@inputs) {
        $input->{type} = $type if defined $input;
    }
```
[31.1378]
[58.910]
```
    $_->{type} = $type foreach @inputs;
```
Insertion in src/lib/Hydra/Helper/AddBuilds.pm at line 203 [13.2663]
[12.1358]
[12.1358]

Insertion in src/lib/Hydra/Helper/AddBuilds.pm at line 218 [13.2663]

[60.3652]

            (defined $input->{revNumber} ? "; rev = " . $input->{revNumber} . "" : "") .

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 246 [13.2663]

B:BD[12.1790] → [61.0:68]

                when (["path", "build", "git", "hg", "sysbuild"]) {

[12.1790]

[62.2236]

                default {

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 248 [13.2663]

∅:D[62.2321] → [12.2227:2245]

B:BD[12.2227] → [12.2227:2245]

B:BD[12.2245] → [61.69:423]

                }
                when (["svn", "svn-checkout", "bzr", "bzr-checkout"]) {
                    push @res, "--arg", $input, (
                        "{ outPath = builtins.storePath " . $alt->{storePath} . "" .
                        (defined $alt->{revision} ? "; rev = " . $alt->{revision} . "" : "") .
                        ";}"
                    );

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 255 [13.2663]
B:BD[12.2295] → [12.2295:2296]
Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 256 [13.2663]
∅:D[53.20] → [12.2297:2323]
B:BD[12.2297] → [12.2297:2323]
B:BD[12.2323] → [36.1347:1377]
B:BD[63.398] → [63.398:450]
```
sub captureStdoutStderr {
    my ($timeout, @cmd) = @_;
    my $stdin = "";
    my $stdout;
    my $stderr;
```
Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 257 [13.2663]
B:BD[63.451] → [63.451:556]
B:BD[63.556] → [4.2969:3028]
∅:D[4.3028] → [63.623:640]
B:BD[63.623] → [63.623:640]
B:BD[63.640] → [64.0:7]
∅:D[64.7] → [63.648:649]
B:BD[63.648] → [63.648:649]
B:BD[63.649] → [64.8:22]
B:BD[64.22] → [4.3029:3135]
∅:D[34.1605] → [64.116:129]
∅:D[4.3135] → [64.116:129]
∅:D[38.3583] → [64.116:129]
B:BD[64.116] → [64.116:129]
B:BD[64.129] → [4.3136:3175]
∅:D[34.1647] → [64.163:169]
∅:D[4.3175] → [64.163:169]
B:BD[64.163] → [64.163:169]
∅:D[64.169] → [63.835:837]
B:BD[63.835] → [63.835:837]
B:BD[63.837] → [53.21:22]
B:BD[53.22] → [36.1378:1379]
```
    eval {
        local $SIG{ALRM} = sub { die "timeout\n" }; # NB: \n required
        alarm $timeout;
        IPC::Run::run(\@cmd, \$stdin, \$stdout, \$stderr);
        alarm 0;
    };
    if ($@) {
        die unless $@ eq "timeout\n"; # propagate unexpected errors
        return (-1, "", "timeout\n");
    } else {
        return ($?, $stdout, $stderr);
    }
}
```
Insertion in src/lib/Hydra/Helper/Nix.pm at line 14 [67.436]
[65.39]
[66.665]
```
    getSCMCacheDir
```
Replacement in src/lib/Hydra/Helper/Nix.pm at line 22 [67.436]
B:BD[5.478] → [5.478:503]
```
    pathIsInsidePrefix);
```
[5.478]
[68.77]
```
    pathIsInsidePrefix
    captureStdoutStderr);
```

Insertion in src/lib/Hydra/Helper/Nix.pm at line 52 [67.436]

[69.935]

}
sub getSCMCacheDir {
    return Hydra::Model::DB::getHydraPath . "/scm" ;

Insertion in src/lib/Hydra/Helper/Nix.pm at line 451 [67.436]
[70.9704]
[70.9704]

Insertion in src/lib/Hydra/Helper/Nix.pm at line 453 [67.436]

[70.9705]

sub captureStdoutStderr {
    my ($timeout, @cmd) = @_;
    my $stdin = "";
    my $stdout;
    my $stderr;

Insertion in src/lib/Hydra/Helper/Nix.pm at line 459 [67.436]

[70.9706]

[71.850]

    eval {
        local $SIG{ALRM} = sub { die "timeout\n" }; # NB: \n required
        alarm $timeout;
        IPC::Run::run(\@cmd, \$stdin, \$stdout, \$stderr);
        alarm 0;
    };
    if ($@) {
        die unless $@ eq "timeout\n"; # propagate unexpected errors
        return (-1, "", "timeout\n");
    } else {
        return ($?, $stdout, $stderr);
    }
}

File addition: BazaarInput.pm (----------)

[72.1]

package Hydra::Plugin::BazaarInput;
use strict;
use parent 'Hydra::Plugin';
use Digest::SHA qw(sha256_hex);
use File::Path;
use Hydra::Helper::Nix;
use Nix::Store;
sub supportedInputTypes {
    my ($self, $inputTypes) = @_;
    $inputTypes->{'bzr'} = 'Bazaar export';
    $inputTypes->{'bzr-checkout'} = 'Bazaar checkout';
}
sub fetchInput {
    my ($self, $type, $name, $value) = @_;
    return undef if $type ne "bzr" && $type ne "bzr-checkout";
    my $uri = $value;
    my $sha256;
    my $storePath;
    my $stdout; my $stderr;
    my $cacheDir = getSCMCacheDir . "/bzr";
    mkpath($cacheDir);
    my $clonePath = $cacheDir . "/" . sha256_hex($uri);
    if (! -d $clonePath) {
        (my $res, $stdout, $stderr) = captureStdoutStderr(600, "bzr", "branch", $uri, $clonePath);
        die "error cloning bazaar branch at `$uri':\n$stderr" if $res;
    }
    chdir $clonePath or die $!;
    (my $res, $stdout, $stderr) = captureStdoutStderr(600, "bzr", "pull");
    die "error pulling latest change bazaar branch at `$uri':\n$stderr" if $res;
    # First figure out the last-modified revision of the URI.
    my @cmd = (["bzr", "revno"], "|", ["sed", 's/^ *\([0-9]*\).*/\1/']);
    IPC::Run::run(@cmd, \$stdout, \$stderr);
    die "cannot get head revision of Bazaar branch at `$uri':\n$stderr" if $?;
    my $revision = $stdout; chomp $revision;
    die unless $revision =~ /^\d+$/;
    (my $cachedInput) = $self->{db}->resultset('CachedBazaarInputs')->search(
        {uri => $uri, revision => $revision});
    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
        $storePath = $cachedInput->storepath;
        $sha256 = $cachedInput->sha256hash;
    } else {
        # Then download this revision into the store.
        print STDERR "checking out Bazaar input ", $name, " from $uri revision $revision\n";
        $ENV{"NIX_HASH_ALGO"} = "sha256";
        $ENV{"PRINT_PATH"} = "1";
        $ENV{"NIX_PREFETCH_BZR_LEAVE_DOT_BZR"} = $type eq "bzr-checkout" ? "1" : "0";
        (my $res, $stdout, $stderr) = captureStdoutStderr(600,
            "nix-prefetch-bzr", $clonePath, $revision);
        die "cannot check out Bazaar branch `$uri':\n$stderr" if $res;
        ($sha256, $storePath) = split ' ', $stdout;
        txn_do($self->{db}, sub {
            $self->{db}->resultset('CachedBazaarInputs')->create(
                { uri => $uri
                , revision => $revision
                , sha256hash => $sha256
                , storepath => $storePath
                });
            });
    }
    return
        { uri => $uri
        , storePath => $storePath
        , sha256hash => $sha256
        , revision => $revision
        };
}
1;

File addition: GitInput.pm (----------)

[72.1]

package Hydra::Plugin::GitInput;
use strict;
use parent 'Hydra::Plugin';
use Digest::SHA qw(sha256_hex);
use File::Path;
use Hydra::Helper::Nix;
use Nix::Store;
sub supportedInputTypes {
    my ($self, $inputTypes) = @_;
    $inputTypes->{'git'} = 'Git checkout';
}
sub fetchInput {
    my ($self, $type, $name, $value) = @_;
    return undef if $type ne "git";
    (my $uri, my $branch, my $deepClone) = split ' ', $value;
    $branch = defined $branch ? $branch : "master";
    my $timestamp = time;
    my $sha256;
    my $storePath;
    my $cacheDir = getSCMCacheDir . "/git";
    mkpath($cacheDir);
    my $clonePath = $cacheDir . "/" . sha256_hex($uri);
    my $stdout = ""; my $stderr = ""; my $res;
    if (! -d $clonePath) {
        # Clone everything and fetch the branch.
        # TODO: Optimize the first clone by using "git init $clonePath" and "git remote add origin $uri".
        ($res, $stdout, $stderr) = captureStdoutStderr(600, "git", "clone", "--branch", $branch, $uri, $clonePath);
        die "error cloning git repo at `$uri':\n$stderr" if $res;
    }
    chdir $clonePath or die $!; # !!! urgh, shouldn't do a chdir
    # This command force the update of the local branch to be in the same as
    # the remote branch for whatever the repository state is.  This command mirror
    # only one branch of the remote repository.
    ($res, $stdout, $stderr) = captureStdoutStderr(600,
        "git", "fetch", "-fu", "origin", "+$branch:$branch");
    ($res, $stdout, $stderr) = captureStdoutStderr(600,
        "git", "fetch", "-fu", "origin") if $res;
    die "error fetching latest change from git repo at `$uri':\n$stderr" if $res;
    ($res, $stdout, $stderr) = captureStdoutStderr(600,
        ("git", "rev-parse", "$branch"));
    die "error getting revision number of Git branch '$branch' at `$uri':\n$stderr" if $res;
    my ($revision) = split /\n/, $stdout;
    die "error getting a well-formated revision number of Git branch '$branch' at `$uri':\n$stdout"
        unless $revision =~ /^[0-9a-fA-F]+$/;
    my $ref = "refs/heads/$branch";
    # If deepClone is defined, then we look at the content of the repository
    # to determine if this is a top-git branch.
    if (defined $deepClone) {
        # Checkout the branch to look at its content.
        ($res, $stdout, $stderr) = captureStdoutStderr(600, "git", "checkout", "$branch");
        die "error checking out Git branch '$branch' at `$uri':\n$stderr" if $res;
        if (-f ".topdeps") {
            # This is a TopGit branch.  Fetch all the topic branches so
            # that builders can run "tg patch" and similar.
            ($res, $stdout, $stderr) = captureStdoutStderr(600,
                "tg", "remote", "--populate", "origin");
            print STDERR "warning: `tg remote --populate origin' failed:\n$stderr" if $res;
        }
    }
    # Some simple caching: don't check a uri/branch/revision more than once.
    # TODO: Fix case where the branch is reset to a previous commit.
    my $cachedInput ;
    ($cachedInput) = $self->{db}->resultset('CachedGitInputs')->search(
        {uri => $uri, branch => $branch, revision => $revision},
        {rows => 1});
    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
        $storePath = $cachedInput->storepath;
        $sha256 = $cachedInput->sha256hash;
        $revision = $cachedInput->revision;
    } else {
        # Then download this revision into the store.
        print STDERR "checking out Git branch $branch from $uri\n";
        $ENV{"NIX_HASH_ALGO"} = "sha256";
        $ENV{"PRINT_PATH"} = "1";
        $ENV{"NIX_PREFETCH_GIT_LEAVE_DOT_GIT"} = "0";
        $ENV{"NIX_PREFETCH_GIT_DEEP_CLONE"} = "";
        if (defined $deepClone) {
            # Checked out code often wants to be able to run `git
            # describe', e.g., code that uses Gnulib's `git-version-gen'
            # script.  Thus, we leave `.git' in there.  Same for
            # Subversion (e.g., libgcrypt's build system uses that.)
            $ENV{"NIX_PREFETCH_GIT_LEAVE_DOT_GIT"} = "1";
            # Ask for a "deep clone" to allow "git describe" and similar
            # tools to work.  See
            # http://thread.gmane.org/gmane.linux.distributions.nixos/3569
            # for a discussion.
            $ENV{"NIX_PREFETCH_GIT_DEEP_CLONE"} = "1";
        }
        ($res, $stdout, $stderr) = captureStdoutStderr(600, "nix-prefetch-git", $clonePath, $revision);
        die "cannot check out Git repository branch '$branch' at `$uri':\n$stderr" if $res;
        ($sha256, $storePath) = split ' ', $stdout;
        txn_do($self->{db}, sub {
            $self->{db}->resultset('CachedGitInputs')->update_or_create(
                { uri => $uri
                , branch => $branch
                , revision => $revision
                , sha256hash => $sha256
                , storepath => $storePath
                });
            });
    }
    # For convenience in producing readable version names, pass the
    # number of commits in the history of this revision (‘revCount’)
    # the output of git-describe (‘gitTag’), and the abbreviated
    # revision (‘shortRev’).
    my $revCount = `git rev-list $revision | wc -l`; chomp $revCount;
    die "git rev-list failed" if $? != 0;
    my $gitTag = `git describe --always $revision`; chomp $gitTag;
    die "git describe failed" if $? != 0;
    my $shortRev = `git rev-parse --short $revision`; chomp $shortRev;
    die "git rev-parse failed" if $? != 0;
    return
        { uri => $uri
        , storePath => $storePath
        , sha256hash => $sha256
        , revision => $revision
        , revCount => int($revCount)
        , gitTag => $gitTag
        , shortRev => $shortRev
        };
}
1;

File addition: MercurialInput.pm (----------)

[72.1]

package Hydra::Plugin::MercurialInput;
use strict;
use parent 'Hydra::Plugin';
use Digest::SHA qw(sha256_hex);
use File::Path;
use Hydra::Helper::Nix;
use Nix::Store;
sub supportedInputTypes {
    my ($self, $inputTypes) = @_;
    $inputTypes->{'hg'} = 'Mercurial checkout';
}
sub fetchInput {
    my ($self, $type, $name, $value) = @_;
    return undef if $type ne "hg";
    (my $uri, my $id) = split ' ', $value;
    $id = defined $id ? $id : "default";
    # init local hg clone
    my $stdout = ""; my $stderr = "";
    my $cacheDir = getSCMCacheDir . "/hg";
    mkpath($cacheDir);
    my $clonePath = $cacheDir . "/" . sha256_hex($uri);
    if (! -d $clonePath) {
        (my $res, $stdout, $stderr) = captureStdoutStderr(600,
            "hg", "clone", $uri, $clonePath);
        die "error cloning mercurial repo at `$uri':\n$stderr" if $res;
    }
    # hg pull + check rev
    chdir $clonePath or die $!;
    (my $res, $stdout, $stderr) = captureStdoutStderr(600, "hg", "pull");
    die "error pulling latest change mercurial repo at `$uri':\n$stderr" if $res;
    (my $res1, $stdout, $stderr) = captureStdoutStderr(600,
        "hg", "log", "-r", $id, "--template", "{node|short} {rev} {branch}");
    die "error getting branch and revision of $id from `$uri':\n$stderr" if $res1;
    my ($revision, $revCount, $branch) = split ' ', $stdout;
    my $storePath;
    my $sha256;
    (my $cachedInput) = $self->{db}->resultset('CachedHgInputs')->search(
        {uri => $uri, branch => $branch, revision => $revision});
    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
        $storePath = $cachedInput->storepath;
        $sha256 = $cachedInput->sha256hash;
    } else {
        print STDERR "checking out Mercurial input from $uri $branch revision $revision\n";
        $ENV{"NIX_HASH_ALGO"} = "sha256";
        $ENV{"PRINT_PATH"} = "1";
        (my $res, $stdout, $stderr) = captureStdoutStderr(600,
            "nix-prefetch-hg", $clonePath, $revision);
        die "cannot check out Mercurial repository `$uri':\n$stderr" if $res;
        ($sha256, $storePath) = split ' ', $stdout;
        txn_do($self->{db}, sub {
            $self->{db}->resultset('CachedHgInputs')->update_or_create(
                { uri => $uri
                , branch => $branch
                , revision => $revision
                , sha256hash => $sha256
                , storepath => $storePath
                });
            });
    }
    return
        { uri => $uri
        , branch => $branch
        , storePath => $storePath
        , sha256hash => $sha256
        , revision => $revision
        , revCount => int($revCount)
        };
}
1;

File addition: PathInput.pm (----------)

[72.1]

package Hydra::Plugin::PathInput;
use strict;
use parent 'Hydra::Plugin';
use POSIX qw(strftime);
use Hydra::Helper::Nix;
use Nix::Store;
sub supportedInputTypes {
    my ($self, $inputTypes) = @_;
    $inputTypes->{'path'} = 'Local path';
}
sub fetchInput {
    my ($self, $type, $name, $value) = @_;
    return undef if $type ne "path";
    my $uri = $value;
    my $timestamp = time;
    my $sha256;
    my $storePath;
    # Some simple caching: don't check a path more than once every N seconds.
    (my $cachedInput) = $self->{db}->resultset('CachedPathInputs')->search(
        {srcpath => $uri, lastseen => {">", $timestamp - 30}},
        {rows => 1, order_by => "lastseen DESC"});
    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
        $storePath = $cachedInput->storepath;
        $sha256 = $cachedInput->sha256hash;
        $timestamp = $cachedInput->timestamp;
    } else {
        print STDERR "copying input ", $name, " from $uri\n";
        $storePath = `nix-store --add "$uri"`
            or die "cannot copy path $uri to the Nix store.\n";
        chomp $storePath;
        $sha256 = (queryPathInfo($storePath, 0))[1] or die;
        ($cachedInput) = $self->{db}->resultset('CachedPathInputs')->search(
            {srcpath => $uri, sha256hash => $sha256});
        # Path inputs don't have a natural notion of a "revision", so
        # we simulate it by using the timestamp that we first saw this
        # path have this SHA-256 hash.  So if the contents of the path
        # changes, we get a new "revision", but if it doesn't change
        # (or changes back), we don't get a new "revision".
        if (!defined $cachedInput) {
            txn_do($self->{db}, sub {
                $self->{db}->resultset('CachedPathInputs')->update_or_create(
                    { srcpath => $uri
                    , timestamp => $timestamp
                    , lastseen => $timestamp
                    , sha256hash => $sha256
                    , storepath => $storePath
                    });
                });
        } else {
            $timestamp = $cachedInput->timestamp;
            txn_do($self->{db}, sub {
                $cachedInput->update({lastseen => time});
            });
        }
    }
    return
        { uri => $uri
        , storePath => $storePath
        , sha256hash => $sha256
        , revision => strftime "%Y%m%d%H%M%S", gmtime($timestamp)
        };
}
1;

File addition: SubversionInput.pm (----------)

[72.1]

package Hydra::Plugin::SubversionInput;
use strict;
use parent 'Hydra::Plugin';
use Digest::SHA qw(sha256_hex);
use Hydra::Helper::Nix;
use IPC::Run;
use Nix::Store;
sub supportedInputTypes {
    my ($self, $inputTypes) = @_;
    $inputTypes->{'svn'} = 'Subversion export';
    $inputTypes->{'svn-checkout'} = 'Subversion checkout';
}
sub fetchInput {
    my ($self, $type, $name, $value) = @_;
    return undef if $type ne "svn" && $type ne "svn-checkout";
    # Allow users to specify a revision number next to the URI.
    my ($uri, $revision) = split ' ', $value;
    my $sha256;
    my $storePath;
    my $stdout; my $stderr;
    unless (defined $revision) {
        # First figure out the last-modified revision of the URI.
        my @cmd = (["svn", "ls", "-v", "--depth", "empty", $uri],
                   "|", ["sed", 's/^ *\([0-9]*\).*/\1/']);
        IPC::Run::run(@cmd, \$stdout, \$stderr);
        die "cannot get head revision of Subversion repository at `$uri':\n$stderr" if $?;
        $revision = int($stdout); $revision =~ s/\s*([0-9]+)\s*/$1/sm;
    }
    die unless $revision =~ /^\d+$/;
    $revision = int($revision);
    # Do we already have this revision in the store?
    # !!! This needs to take $checkout into account!  Otherwise "svn"
    # and "svn-checkout" inputs can get mixed up.
    (my $cachedInput) = $self->{db}->resultset('CachedSubversionInputs')->search(
        {uri => $uri, revision => $revision});
    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
        $storePath = $cachedInput->storepath;
        $sha256 = $cachedInput->sha256hash;
    } else {
        # No, do a checkout.  The working copy is reused between
        # invocations to speed things up.
        my $wcPath = getSCMCacheDir . "/svn/" . sha256_hex($uri) . "/svn-checkout";
        print STDERR "checking out Subversion input ", $name, " from $uri revision $revision into $wcPath\n";
        (my $res, $stdout, $stderr) = captureStdoutStderr(600, "svn", "checkout", $uri, "-r", $revision, $wcPath);
        die "error checking out Subversion repo at `$uri':\n$stderr" if $res;
        if ($type eq "svn-checkout") {
            $storePath = addToStore($wcPath, 1, "sha256");
        } else {
            # Hm, if the Nix Perl bindings supported filters in
            # addToStore(), then we wouldn't need to make a copy here.
            my $tmpDir = File::Temp->newdir("hydra-svn-export.XXXXXX", CLEANUP => 1, TMPDIR => 1) or die;
            (system "svn", "export", $wcPath, "$tmpDir/svn-export", "--quiet") == 0 or die "svn export failed";
            $storePath = addToStore("$tmpDir/svn-export", 1, "sha256");
        }
        $sha256 = queryPathHash($storePath); $sha256 =~ s/sha256://;
        txn_do($self->{db}, sub {
            $self->{db}->resultset('CachedSubversionInputs')->update_or_create(
                { uri => $uri
                , revision => $revision
                , sha256hash => $sha256
                , storepath => $storePath
                });
            });
    }
    return
        { uri => $uri
        , storePath => $storePath
        , sha256hash => $sha256
        , revNumber => $revision
        };
}
1;

Insertion in src/lib/Hydra/Plugin.pm at line 21 [72.5909]

[6.523]

}
# Called to determine the set of supported input types.  The plugin
# should add these to the $inputTypes hashref, e.g. $inputTypes{'svn'}
# = 'Subversion checkout'.
sub supportedInputTypes {
    my ($self, $inputTypes) = @_;

Insertion in src/lib/Hydra/Plugin.pm at line 30 [72.5909]

[72.6305]

# Called to fetch an input of type ‘$type’.  ‘$value’ is the input
# location, typically the repository URL.
sub fetchInput {
    my ($self, $type, $name, $value) = @_;
    return undef;
}

Insertion in src/lib/Hydra.pm at line 7 [16.6954]
[7.60]
[73.0]
```
use Hydra::Plugin;
```
Insertion in src/lib/Hydra.pm at line 85 [16.6954]
[7.179]
[7.179]
```
my $plugins;
```

Insertion in src/lib/Hydra.pm at line 88 [16.6954]

[7.180]

[16.7263]

has 'hydra_plugins' => (
    is => 'ro',
    default => sub { return $plugins; }
);
after setup_finalize => sub {
    my $class = shift;
    $plugins = [Hydra::Plugin->plugins(db => $class->model('DB'), config => $class->config)];
};

Deletion in src/root/common.tt at line 5 [76.2803]
∅:D[74.35] → [75.694:695]
∅:D[77.53] → [75.694:695]
∅:D[78.588] → [75.694:695]
∅:D[79.6935] → [75.694:695]
B:BD[75.694] → [75.694:695]

Deletion in src/root/common.tt at line 6 [76.2803]

B:BD[75.696] → [74.36:412]

inputTypes =
  { "svn" = "Subversion export"
  , "svn-checkout" = "Subversion checkout"
  , "bzr" = "Bazaar export"
  , "bzr-checkout" = "Bazaar checkout"
  , "git" = "Git checkout"
  , "hg" = "Mercurial checkout"
  , "string" = "String value"
  , "boolean" = "Boolean"
  , "path" = "Local path"
  , "build" = "Build output"
  , "sysbuild" = "Build output (same system)"
  };

Deletion in src/root/edit-jobset.tt at line 170 [80.367]
B:BD[80.5703] → [80.5703:5705]
Insertion in src/script/hydra-evaluator at line 6 [82.1319]
[81.492]
[71.1746]
```
use Hydra::Plugin;
```

Insertion in src/script/hydra-evaluator at line 24 [82.1319]

[83.1]

my $plugins = [Hydra::Plugin->plugins(db => $db, config => $config)];

Replacement in src/script/hydra-evaluator at line 35 [82.1319]

B:BD[84.646] → [65.993:1230]

            my @info = fetchInput($db, $project, $jobset, $input->name, $input->type, $alt->value);
            foreach my $info_el (@info) {
                push @{$$inputInfo{$input->name}}, $info_el if defined $info_el;
            }

[84.646]

[84.722]

            push @{$$inputInfo{$input->name}}, $_
                foreach fetchInput($plugins, $db, $project, $jobset, $input->name, $input->type, $alt->value);

Support revision control systems via plugins

Dependencies

In channels

Change contents

Insertion in src/lib/Hydra/Controller/API.pm at line 221 [10.24]

Replacement in src/lib/Hydra/Controller/API.pm at line 226 [10.24]

Replacement in src/lib/Hydra/Controller/API.pm at line 232 [10.24]

Replacement in src/lib/Hydra/Controller/Build.pm at line 536 [14.1]

Insertion in src/lib/Hydra/Controller/Root.pm at line 31 [16.804]

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 6 [13.2663]

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 21 [13.2663]

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 25 [13.2663]

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 72 [13.2663]

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 127 [13.2663]

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 150 [13.2663]

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 151 [13.2663]

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 152 [13.2663]

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 153 [13.2663]

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 156 [13.2663]

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 160 [13.2663]

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 162 [13.2663]

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 164 [13.2663]

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 168 [13.2663]

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 171 [13.2663]

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 182 [13.2663]

Insertion in src/lib/Hydra/Helper/AddBuilds.pm at line 203 [13.2663]

Insertion in src/lib/Hydra/Helper/AddBuilds.pm at line 218 [13.2663]

Replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 246 [13.2663]

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 248 [13.2663]

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 255 [13.2663]

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 256 [13.2663]

Deletion in src/lib/Hydra/Helper/AddBuilds.pm at line 257 [13.2663]

Insertion in src/lib/Hydra/Helper/Nix.pm at line 14 [67.436]

Replacement in src/lib/Hydra/Helper/Nix.pm at line 22 [67.436]

Insertion in src/lib/Hydra/Helper/Nix.pm at line 52 [67.436]

Insertion in src/lib/Hydra/Helper/Nix.pm at line 451 [67.436]

Insertion in src/lib/Hydra/Helper/Nix.pm at line 453 [67.436]

Insertion in src/lib/Hydra/Helper/Nix.pm at line 459 [67.436]

File addition: BazaarInput.pm (----------)

File addition: GitInput.pm (----------)

File addition: MercurialInput.pm (----------)

File addition: PathInput.pm (----------)

File addition: SubversionInput.pm (----------)

Insertion in src/lib/Hydra/Plugin.pm at line 21 [72.5909]

Insertion in src/lib/Hydra/Plugin.pm at line 30 [72.5909]

Insertion in src/lib/Hydra.pm at line 7 [16.6954]

Insertion in src/lib/Hydra.pm at line 85 [16.6954]

Insertion in src/lib/Hydra.pm at line 88 [16.6954]

Deletion in src/root/common.tt at line 5 [76.2803]

Deletion in src/root/common.tt at line 6 [76.2803]

Deletion in src/root/edit-jobset.tt at line 170 [80.367]

Insertion in src/script/hydra-evaluator at line 6 [82.1319]

Insertion in src/script/hydra-evaluator at line 24 [82.1319]

Replacement in src/script/hydra-evaluator at line 35 [82.1319]