mercurial-source: mercurial/bundle2.py annotate

annotate mercurial/bundle2.py @ 34671:891118dcd279

bundle2: seek part back during iteration Previously, iterparts would yield the part to users, then consume the part. This changed the part after the user was given it and left it at the end, both of which seem unexpected. Let's seek back to the beginning after we've consumed it. I tried not seeking to the end at all, but that seems important for the overall bundle2 consumption. This is used in a future patch to let us move the bundlerepo bundle2-changegroup-part to be handled entirely within the for loop, instead of having to do a seek back to 0 after the entire loop finishes. Differential Revision: https://phab.mercurial-scm.org/D289

author	Durham Goode <durham@fb.com>
date	Wed, 23 Aug 2017 12:35:03 -0700
parents	13dc7f29531e
children	5ede882c249c

rev	line source
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	1 # bundle2.py - generic container format to transmit arbitrary data.
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	2 #
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	3 # Copyright 2013 Facebook, Inc.
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	4 #
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	5 # This software may be used and distributed according to the terms of the
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	6 # GNU General Public License version 2 or any later version.
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	7 """Handling of the new bundle2 format
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	8
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	9 The goal of bundle2 is to act as an atomically packet to transmit a set of
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	10 payloads in an application agnostic way. It consist in a sequence of "parts"
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	11 that will be handed to and processed by the application layer.
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	12
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	13
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	14 General format architecture
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	15 ===========================
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	16
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	17 The format is architectured as follow
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	18
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	19 - magic string
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	20 - stream level parameters
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	21 - payload parts (any number)
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	22 - end of stream marker.
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	23
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	24 the Binary format
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	25 ============================
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	26
21024 7731a2281cf0 spelling: fixes from spell checker Mads Kiilerich <madski@unity3d.com> parents: 21020 diff changeset	27 All numbers are unsigned and big-endian.
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	28
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	29 stream level parameters
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	30 ------------------------
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	31
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	32 Binary format is as follow
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	33
23009 90f86ad3d4ff bundle2: change header size and make them signed (new format) Pierre-Yves David <pierre-yves.david@fb.com> parents: 23008 diff changeset	34 :params size: int32
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	35
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	36 The total number of Bytes used by the parameters
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	37
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	38 :params value: arbitrary number of Bytes
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	39
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	40 A blob of `params size` containing the serialized version of all stream level
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	41 parameters.
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	42
21024 7731a2281cf0 spelling: fixes from spell checker Mads Kiilerich <madski@unity3d.com> parents: 21020 diff changeset	43 The blob contains a space separated list of parameters. Parameters with value
20811 9785c3f8f598 bundle2: urlquote stream parameter name and value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20810 diff changeset	44 are stored in the form `<name>=<value>`. Both name and value are urlquoted.
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	45
20813 8c74b3ce5b70 bundle2: refuse empty parameter name Pierre-Yves David <pierre-yves.david@fb.com> parents: 20812 diff changeset	46 Empty name are obviously forbidden.
8c74b3ce5b70 bundle2: refuse empty parameter name Pierre-Yves David <pierre-yves.david@fb.com> parents: 20812 diff changeset	47
20844 2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	48 Name MUST start with a letter. If this first letter is lower case, the
21024 7731a2281cf0 spelling: fixes from spell checker Mads Kiilerich <madski@unity3d.com> parents: 21020 diff changeset	49 parameter is advisory and can be safely ignored. However when the first
20844 2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	50 letter is capital, the parameter is mandatory and the bundling process MUST
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	51 stop if he is not able to proceed it.
20814 8532f5e1b9df bundle2: force the first char of parameter to be an letter. Pierre-Yves David <pierre-yves.david@fb.com> parents: 20813 diff changeset	52
20808 4c9130c7a29f bundle2: clarify stream parameter design in the documentation Pierre-Yves David <pierre-yves.david@fb.com> parents: 20805 diff changeset	53 Stream parameters use a simple textual format for two main reasons:
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	54
21024 7731a2281cf0 spelling: fixes from spell checker Mads Kiilerich <madski@unity3d.com> parents: 21020 diff changeset	55 - Stream level parameters should remain simple and we want to discourage any
20808 4c9130c7a29f bundle2: clarify stream parameter design in the documentation Pierre-Yves David <pierre-yves.david@fb.com> parents: 20805 diff changeset	56 crazy usage.
21024 7731a2281cf0 spelling: fixes from spell checker Mads Kiilerich <madski@unity3d.com> parents: 21020 diff changeset	57 - Textual data allow easy human inspection of a bundle2 header in case of
20808 4c9130c7a29f bundle2: clarify stream parameter design in the documentation Pierre-Yves David <pierre-yves.david@fb.com> parents: 20805 diff changeset	58 troubles.
4c9130c7a29f bundle2: clarify stream parameter design in the documentation Pierre-Yves David <pierre-yves.david@fb.com> parents: 20805 diff changeset	59
4c9130c7a29f bundle2: clarify stream parameter design in the documentation Pierre-Yves David <pierre-yves.david@fb.com> parents: 20805 diff changeset	60 Any Applicative level options MUST go into a bundle2 part instead.
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	61
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	62 Payload part
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	63 ------------------------
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	64
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	65 Binary format is as follow
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	66
23009 90f86ad3d4ff bundle2: change header size and make them signed (new format) Pierre-Yves David <pierre-yves.david@fb.com> parents: 23008 diff changeset	67 :header size: int32
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	68
26043 5bee483705e8 bundle2: clarify in docstring that header size is for a single header Martin von Zweigbergk <martinvonz@google.com> parents: 26042 diff changeset	69 The total number of Bytes used by the part header. When the header is empty
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	70 (size = 0) this is interpreted as the end of stream marker.
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	71
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	72 :header:
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	73
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	74 The header defines how to interpret the part. It contains two piece of
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	75 data: the part type, and the part parameters.
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	76
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	77 The part type is used to route an application level handler, that can
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	78 interpret payload.
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	79
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	80 Part parameters are passed to the application level handler. They are
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	81 meant to convey information that will help the application level object to
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	82 interpret the part payload.
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	83
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	84 The binary format of the header is has follow
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	85
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	86 :typesize: (one byte)
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	87
23928 a3f7c781786b bundle2: fix parttype enforcement Matt Mackall <mpm@selenic.com> parents: 23880 diff changeset	88 :parttype: alphanumerical part name (restricted to [a-zA-Z0-9_:-]*)
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	89
20995 e995d104c87f bundle2: add an integer id to part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20952 diff changeset	90 :partid: A 32bits integer (unique in the bundle) that can be used to refer
e995d104c87f bundle2: add an integer id to part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20952 diff changeset	91 to this part.
e995d104c87f bundle2: add an integer id to part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20952 diff changeset	92
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	93 :parameters:
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	94
21024 7731a2281cf0 spelling: fixes from spell checker Mads Kiilerich <madski@unity3d.com> parents: 21020 diff changeset	95 Part's parameter may have arbitrary content, the binary structure is::
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	96
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	97 <mandatory-count><advisory-count><param-sizes><param-data>
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	98
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	99 :mandatory-count: 1 byte, number of mandatory parameters
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	100
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	101 :advisory-count: 1 byte, number of advisory parameters
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	102
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	103 :param-sizes:
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	104
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	105 N couple of bytes, where N is the total number of parameters. Each
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	106 couple contains (<size-of-key>, <size-of-value) for one parameter.
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	107
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	108 :param-data:
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	109
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	110 A blob of bytes from which each parameter key and value can be
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	111 retrieved using the list of size couples stored in the previous
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	112 field.
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	113
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	114 Mandatory parameters comes first, then the advisory ones.
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	115
21607 054fa5176fa7 bundle2: forbid duplicate parameter keys Pierre-Yves David <pierre-yves.david@fb.com> parents: 21606 diff changeset	116 Each parameter's key MUST be unique within the part.
054fa5176fa7 bundle2: forbid duplicate parameter keys Pierre-Yves David <pierre-yves.david@fb.com> parents: 21606 diff changeset	117
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	118 :payload:
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	119
20876 ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	120 payload is a series of `<chunksize><chunkdata>`.
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	121
23009 90f86ad3d4ff bundle2: change header size and make them signed (new format) Pierre-Yves David <pierre-yves.david@fb.com> parents: 23008 diff changeset	122 `chunksize` is an int32, `chunkdata` are plain bytes (as much as
20876 ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	123 `chunksize` says)` The payload part is concluded by a zero size chunk.
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	124
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	125 The current implementation always produces either zero or one chunk.
21024 7731a2281cf0 spelling: fixes from spell checker Mads Kiilerich <madski@unity3d.com> parents: 21020 diff changeset	126 This is an implementation limitation that will ultimately be lifted.
20891 1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	127
23009 90f86ad3d4ff bundle2: change header size and make them signed (new format) Pierre-Yves David <pierre-yves.david@fb.com> parents: 23008 diff changeset	128 `chunksize` can be negative to trigger special case processing. No such
90f86ad3d4ff bundle2: change header size and make them signed (new format) Pierre-Yves David <pierre-yves.david@fb.com> parents: 23008 diff changeset	129 processing is in place yet.
90f86ad3d4ff bundle2: change header size and make them signed (new format) Pierre-Yves David <pierre-yves.david@fb.com> parents: 23008 diff changeset	130
20891 1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	131 Bundle processing
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	132 ============================
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	133
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	134 Each part is processed in order using a "part handler". Handler are registered
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	135 for a certain part type.
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	136
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	137 The matching of a part to its handler is case insensitive. The case of the
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	138 part type is used to know if a part is mandatory or advisory. If the Part type
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	139 contains any uppercase char it is considered mandatory. When no handler is
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	140 known for a Mandatory part, the process is aborted and an exception is raised.
20892 6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	141 If the part is advisory and no handler is known, the part is ignored. When the
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	142 process is aborted, the full bundle is still read from the stream to keep the
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	143 channel usable. But none of the part read from an abort are processed. In the
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	144 future, dropping the stream may become an option for channel we do not care to
6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	145 preserve.
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	146 """
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	147
34455 da7c285ec6da bundle2: use modern Python division Augie Fackler <augie@google.com> parents: 34448 diff changeset	148 from __future__ import absolute_import, division
26503 8221fefaea08 bundle2: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26214 diff changeset	149
24038 3daef83a1873 bundle2.unpackermixin: control for underlying file descriptor Eric Sumner <ericsumner@fb.com> parents: 23928 diff changeset	150 import errno
26503 8221fefaea08 bundle2: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26214 diff changeset	151 import re
20814 8532f5e1b9df bundle2: force the first char of parameter to be an letter. Pierre-Yves David <pierre-yves.david@fb.com> parents: 20813 diff changeset	152 import string
26503 8221fefaea08 bundle2: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26214 diff changeset	153 import struct
8221fefaea08 bundle2: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26214 diff changeset	154 import sys
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	155
26503 8221fefaea08 bundle2: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26214 diff changeset	156 from .i18n import _
8221fefaea08 bundle2: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26214 diff changeset	157 from . import (
8221fefaea08 bundle2: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26214 diff changeset	158 changegroup,
8221fefaea08 bundle2: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26214 diff changeset	159 error,
8221fefaea08 bundle2: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26214 diff changeset	160 obsolete,
33814 e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	161 phases,
26503 8221fefaea08 bundle2: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26214 diff changeset	162 pushkey,
30810 0f6d6fdd3c2a pycompat: provide 'ispy3' constant Yuya Nishihara <yuya@tcha.org> parents: 30707 diff changeset	163 pycompat,
26503 8221fefaea08 bundle2: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26214 diff changeset	164 tags,
8221fefaea08 bundle2: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26214 diff changeset	165 url,
8221fefaea08 bundle2: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26214 diff changeset	166 util,
8221fefaea08 bundle2: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26214 diff changeset	167 )
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	168
29633 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 29422 diff changeset	169 urlerr = util.urlerr
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 29422 diff changeset	170 urlreq = util.urlreq
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 29422 diff changeset	171
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	172 _pack = struct.pack
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	173 _unpack = struct.unpack
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	174
23009 90f86ad3d4ff bundle2: change header size and make them signed (new format) Pierre-Yves David <pierre-yves.david@fb.com> parents: 23008 diff changeset	175 _fstreamparamsize = '>i'
90f86ad3d4ff bundle2: change header size and make them signed (new format) Pierre-Yves David <pierre-yves.david@fb.com> parents: 23008 diff changeset	176 _fpartheadersize = '>i'
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	177 _fparttypesize = '>B'
20995 e995d104c87f bundle2: add an integer id to part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20952 diff changeset	178 _fpartid = '>I'
23009 90f86ad3d4ff bundle2: change header size and make them signed (new format) Pierre-Yves David <pierre-yves.david@fb.com> parents: 23008 diff changeset	179 _fpayloadsize = '>i'
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	180 _fpartparamcount = '>BB'
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	181
33814 e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	182 _fphasesentry = '>i20s'
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	183
21001 c93bb6a08fa1 bundle2: support chunk iterator as part data Pierre-Yves David <pierre-yves.david@fb.com> parents: 21000 diff changeset	184 preferedchunksize = 4096
c93bb6a08fa1 bundle2: support chunk iterator as part data Pierre-Yves David <pierre-yves.david@fb.com> parents: 21000 diff changeset	185
23880 405eacbecc52 bundle2: enforce parttype as alphanumerical Pierre-Yves David <pierre-yves.david@fb.com> parents: 23602 diff changeset	186 _parttypeforbidden = re.compile('[^a-zA-Z0-9_:-]')
405eacbecc52 bundle2: enforce parttype as alphanumerical Pierre-Yves David <pierre-yves.david@fb.com> parents: 23602 diff changeset	187
25797 8f2c362bcb58 bundle2: introduce a specific function for bundling debug message Pierre-Yves David <pierre-yves.david@fb.com> parents: 25616 diff changeset	188 def outdebug(ui, message):
8f2c362bcb58 bundle2: introduce a specific function for bundling debug message Pierre-Yves David <pierre-yves.david@fb.com> parents: 25616 diff changeset	189 """debug regarding output stream (bundling)"""
33942 0224820688ac config: register the 'devel.bundle2.debug' config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33911 diff changeset	190 if ui.configbool('devel', 'bundle2.debug'):
25820 7c60a42265fb bundle2: hide bundle2 stream debug under a config flag Pierre-Yves David <pierre-yves.david@fb.com> parents: 25819 diff changeset	191 ui.debug('bundle2-output: %s\n' % message)
25797 8f2c362bcb58 bundle2: introduce a specific function for bundling debug message Pierre-Yves David <pierre-yves.david@fb.com> parents: 25616 diff changeset	192
25802 64dd8f25fcf4 bundle2: introduce a specific function for debug messages while unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 25801 diff changeset	193 def indebug(ui, message):
64dd8f25fcf4 bundle2: introduce a specific function for debug messages while unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 25801 diff changeset	194 """debug on input stream (unbundling)"""
33942 0224820688ac config: register the 'devel.bundle2.debug' config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33911 diff changeset	195 if ui.configbool('devel', 'bundle2.debug'):
25820 7c60a42265fb bundle2: hide bundle2 stream debug under a config flag Pierre-Yves David <pierre-yves.david@fb.com> parents: 25819 diff changeset	196 ui.debug('bundle2-input: %s\n' % message)
25802 64dd8f25fcf4 bundle2: introduce a specific function for debug messages while unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 25801 diff changeset	197
23880 405eacbecc52 bundle2: enforce parttype as alphanumerical Pierre-Yves David <pierre-yves.david@fb.com> parents: 23602 diff changeset	198 def validateparttype(parttype):
405eacbecc52 bundle2: enforce parttype as alphanumerical Pierre-Yves David <pierre-yves.david@fb.com> parents: 23602 diff changeset	199 """raise ValueError if a parttype contains invalid character"""
23928 a3f7c781786b bundle2: fix parttype enforcement Matt Mackall <mpm@selenic.com> parents: 23880 diff changeset	200 if _parttypeforbidden.search(parttype):
23880 405eacbecc52 bundle2: enforce parttype as alphanumerical Pierre-Yves David <pierre-yves.david@fb.com> parents: 23602 diff changeset	201 raise ValueError(parttype)
405eacbecc52 bundle2: enforce parttype as alphanumerical Pierre-Yves David <pierre-yves.david@fb.com> parents: 23602 diff changeset	202
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	203 def _makefpartparamsizes(nbparams):
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	204 """return a struct format to read part parameter sizes
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	205
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	206 The number parameters is variable so we need to build that format
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	207 dynamically.
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	208 """
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	209 return '>'+('BB'*nbparams)
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	210
20890 ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	211 parthandlermapping = {}
20889 deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	212
21623 5b26d82e4e2a bundle2: make it possible to declare params handled by a part handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21622 diff changeset	213 def parthandler(parttype, params=()):
20890 ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	214 """decorator that register a function as a bundle2 part handler
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	215
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	216 eg::
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	217
21624 d61066d787c8 bundle2: declare supported parameters for all handlers Pierre-Yves David <pierre-yves.david@fb.com> parents: 21623 diff changeset	218 @parthandler('myparttype', ('mandatory', 'param', 'handled'))
20890 ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	219 def myparttypehandler(...):
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	220 '''process a part of type "my part".'''
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	221 ...
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	222 """
23880 405eacbecc52 bundle2: enforce parttype as alphanumerical Pierre-Yves David <pierre-yves.david@fb.com> parents: 23602 diff changeset	223 validateparttype(parttype)
20890 ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	224 def _decorator(func):
20891 1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	225 lparttype = parttype.lower() # enforce lower case matching.
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	226 assert lparttype not in parthandlermapping
1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	227 parthandlermapping[lparttype] = func
21623 5b26d82e4e2a bundle2: make it possible to declare params handled by a part handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21622 diff changeset	228 func.params = frozenset(params)
20890 ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	229 return func
ec7fc110faee bundle2: introduce a `parthandler` decorator Pierre-Yves David <pierre-yves.david@fb.com> parents: 20889 diff changeset	230 return _decorator
20889 deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	231
20949 571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	232 class unbundlerecords(object):
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	233 """keep record of what happens during and unbundle
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	234
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	235 New records are added using `records.add('cat', obj)`. Where 'cat' is a
21024 7731a2281cf0 spelling: fixes from spell checker Mads Kiilerich <madski@unity3d.com> parents: 21020 diff changeset	236 category of record and obj is an arbitrary object.
20949 571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	237
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	238 `records['cat']` will return all entries of this category 'cat'.
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	239
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	240 Iterating on the object itself will yield `('category', obj)` tuples
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	241 for all entries.
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	242
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	243 All iterations happens in chronological order.
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	244 """
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	245
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	246 def __init__(self):
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	247 self._categories = {}
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	248 self._sequences = []
20996 ed3c5e18a047 bundle2: add reply awareness to unbundlerecords Pierre-Yves David <pierre-yves.david@fb.com> parents: 20995 diff changeset	249 self._replies = {}
20949 571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	250
20996 ed3c5e18a047 bundle2: add reply awareness to unbundlerecords Pierre-Yves David <pierre-yves.david@fb.com> parents: 20995 diff changeset	251 def add(self, category, entry, inreplyto=None):
20949 571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	252 """add a new record of a given category.
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	253
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	254 The entry can then be retrieved in the list returned by
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	255 self['category']."""
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	256 self._categories.setdefault(category, []).append(entry)
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	257 self._sequences.append((category, entry))
20996 ed3c5e18a047 bundle2: add reply awareness to unbundlerecords Pierre-Yves David <pierre-yves.david@fb.com> parents: 20995 diff changeset	258 if inreplyto is not None:
ed3c5e18a047 bundle2: add reply awareness to unbundlerecords Pierre-Yves David <pierre-yves.david@fb.com> parents: 20995 diff changeset	259 self.getreplies(inreplyto).add(category, entry)
ed3c5e18a047 bundle2: add reply awareness to unbundlerecords Pierre-Yves David <pierre-yves.david@fb.com> parents: 20995 diff changeset	260
ed3c5e18a047 bundle2: add reply awareness to unbundlerecords Pierre-Yves David <pierre-yves.david@fb.com> parents: 20995 diff changeset	261 def getreplies(self, partid):
23139 e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues Mads Kiilerich <madski@unity3d.com> parents: 23067 diff changeset	262 """get the records that are replies to a specific part"""
20996 ed3c5e18a047 bundle2: add reply awareness to unbundlerecords Pierre-Yves David <pierre-yves.david@fb.com> parents: 20995 diff changeset	263 return self._replies.setdefault(partid, unbundlerecords())
20949 571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	264
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	265 def __getitem__(self, cat):
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	266 return tuple(self._categories.get(cat, ()))
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	267
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	268 def __iter__(self):
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	269 return iter(self._sequences)
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	270
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	271 def __len__(self):
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	272 return len(self._sequences)
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	273
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	274 def __nonzero__(self):
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	275 return bool(self._sequences)
571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	276
32255 413b44003462 py3: add __bool__ to every class defining __nonzero__ Gregory Szorc <gregory.szorc@gmail.com> parents: 31744 diff changeset	277 __bool__ = __nonzero__
413b44003462 py3: add __bool__ to every class defining __nonzero__ Gregory Szorc <gregory.szorc@gmail.com> parents: 31744 diff changeset	278
20948 329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	279 class bundleoperation(object):
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	280 """an object that represents a single bundling process
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	281
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	282 Its purpose is to carry unbundle-related objects and states.
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	283
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	284 A new object should be created at the beginning of each bundle processing.
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	285 The object is to be returned by the processing function.
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	286
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	287 The object has very little content now it will ultimately contain:
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	288 * an access to the repo the bundle is applied to,
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	289 * a ui object,
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	290 * a way to retrieve a transaction to add changes to the repo,
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	291 * a way to record the result of processing each part,
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	292 * a way to construct a bundle response when applicable.
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	293 """
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	294
25135 e530cde6d115 bundle2: disable ouput capture unless we use http (issue4613 issue4615) Pierre-Yves David <pierre-yves.david@fb.com> parents: 25056 diff changeset	295 def __init__(self, repo, transactiongetter, captureoutput=True):
20948 329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	296 self.repo = repo
329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	297 self.ui = repo.ui
20949 571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	298 self.records = unbundlerecords()
20997 d7df4b7378ae bundle2: produce a bundle2 reply Pierre-Yves David <pierre-yves.david@fb.com> parents: 20996 diff changeset	299 self.reply = None
25135 e530cde6d115 bundle2: disable ouput capture unless we use http (issue4613 issue4615) Pierre-Yves David <pierre-yves.david@fb.com> parents: 25056 diff changeset	300 self.captureoutput = captureoutput
34447 f3407d56a6e8 bundle2: add the capability to store hookargs on bundle operation object Pulkit Goyal <7895pulkit@gmail.com> parents: 34324 diff changeset	301 self.hookargs = {}
34448 5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one Pulkit Goyal <7895pulkit@gmail.com> parents: 34447 diff changeset	302 self._gettransaction = transactiongetter
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one Pulkit Goyal <7895pulkit@gmail.com> parents: 34447 diff changeset	303
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one Pulkit Goyal <7895pulkit@gmail.com> parents: 34447 diff changeset	304 def gettransaction(self):
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one Pulkit Goyal <7895pulkit@gmail.com> parents: 34447 diff changeset	305 transaction = self._gettransaction()
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one Pulkit Goyal <7895pulkit@gmail.com> parents: 34447 diff changeset	306
34593 b3a41f4d837b bundle2: relax the condition to update transaction.hookargs Yuya Nishihara <yuya@tcha.org> parents: 34592 diff changeset	307 if self.hookargs:
34448 5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one Pulkit Goyal <7895pulkit@gmail.com> parents: 34447 diff changeset	308 # the ones added to the transaction supercede those added
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one Pulkit Goyal <7895pulkit@gmail.com> parents: 34447 diff changeset	309 # to the operation.
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one Pulkit Goyal <7895pulkit@gmail.com> parents: 34447 diff changeset	310 self.hookargs.update(transaction.hookargs)
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one Pulkit Goyal <7895pulkit@gmail.com> parents: 34447 diff changeset	311 transaction.hookargs = self.hookargs
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one Pulkit Goyal <7895pulkit@gmail.com> parents: 34447 diff changeset	312
34593 b3a41f4d837b bundle2: relax the condition to update transaction.hookargs Yuya Nishihara <yuya@tcha.org> parents: 34592 diff changeset	313 # mark the hookargs as flushed. further attempts to add to
b3a41f4d837b bundle2: relax the condition to update transaction.hookargs Yuya Nishihara <yuya@tcha.org> parents: 34592 diff changeset	314 # hookargs will result in an abort.
b3a41f4d837b bundle2: relax the condition to update transaction.hookargs Yuya Nishihara <yuya@tcha.org> parents: 34592 diff changeset	315 self.hookargs = None
34448 5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one Pulkit Goyal <7895pulkit@gmail.com> parents: 34447 diff changeset	316
5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one Pulkit Goyal <7895pulkit@gmail.com> parents: 34447 diff changeset	317 return transaction
34447 f3407d56a6e8 bundle2: add the capability to store hookargs on bundle operation object Pulkit Goyal <7895pulkit@gmail.com> parents: 34324 diff changeset	318
f3407d56a6e8 bundle2: add the capability to store hookargs on bundle operation object Pulkit Goyal <7895pulkit@gmail.com> parents: 34324 diff changeset	319 def addhookargs(self, hookargs):
34448 5fc4ddfbe626 bundle2: load hookargs from bundleoperation into transaction when started one Pulkit Goyal <7895pulkit@gmail.com> parents: 34447 diff changeset	320 if self.hookargs is None:
34592 1bf5c5507614 bundle2: raise ProgrammingError for invalid call of addhookargs() Yuya Nishihara <yuya@tcha.org> parents: 34553 diff changeset	321 raise error.ProgrammingError('attempted to add hookargs to '
1bf5c5507614 bundle2: raise ProgrammingError for invalid call of addhookargs() Yuya Nishihara <yuya@tcha.org> parents: 34553 diff changeset	322 'operation after transaction started')
34447 f3407d56a6e8 bundle2: add the capability to store hookargs on bundle operation object Pulkit Goyal <7895pulkit@gmail.com> parents: 34324 diff changeset	323 self.hookargs.update(hookargs)
20948 329cd74b52bd bundle2: introduce a bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20947 diff changeset	324
20952 b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20950 diff changeset	325 class TransactionUnavailable(RuntimeError):
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20950 diff changeset	326 pass
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20950 diff changeset	327
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20950 diff changeset	328 def _notransaction():
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20950 diff changeset	329 """default method to get a transaction while processing a bundle
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20950 diff changeset	330
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20950 diff changeset	331 Raise an exception to highlight the fact that no transaction was expected
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20950 diff changeset	332 to be created"""
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20950 diff changeset	333 raise TransactionUnavailable()
b24ee5076b94 bundle2: make it possible have a global transaction for the unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20950 diff changeset	334
33826 18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1() Martin von Zweigbergk <martinvonz@google.com> parents: 33823 diff changeset	335 def applybundle(repo, unbundler, tr, source=None, url=None, **kwargs):
27430 28a6c2d72097 bundle2: introduce an "applybundle" function Pierre-Yves David <pierre-yves.david@fb.com> parents: 27338 diff changeset	336 # transform me into unbundler.apply() as soon as the freeze is lifted
33826 18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1() Martin von Zweigbergk <martinvonz@google.com> parents: 33823 diff changeset	337 if isinstance(unbundler, unbundle20):
18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1() Martin von Zweigbergk <martinvonz@google.com> parents: 33823 diff changeset	338 tr.hookargs['bundle2'] = '1'
18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1() Martin von Zweigbergk <martinvonz@google.com> parents: 33823 diff changeset	339 if source is not None and 'source' not in tr.hookargs:
18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1() Martin von Zweigbergk <martinvonz@google.com> parents: 33823 diff changeset	340 tr.hookargs['source'] = source
18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1() Martin von Zweigbergk <martinvonz@google.com> parents: 33823 diff changeset	341 if url is not None and 'url' not in tr.hookargs:
18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1() Martin von Zweigbergk <martinvonz@google.com> parents: 33823 diff changeset	342 tr.hookargs['url'] = url
18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1() Martin von Zweigbergk <martinvonz@google.com> parents: 33823 diff changeset	343 return processbundle(repo, unbundler, lambda: tr)
18c2489ac96d bundle: make applybundle() delegate v1 bundles to applybundle1() Martin von Zweigbergk <martinvonz@google.com> parents: 33823 diff changeset	344 else:
33827 1d2b68952950 bundle: inline applybundle1() Martin von Zweigbergk <martinvonz@google.com> parents: 33826 diff changeset	345 # the transactiongetter won't be used, but we might as well set it
1d2b68952950 bundle: inline applybundle1() Martin von Zweigbergk <martinvonz@google.com> parents: 33826 diff changeset	346 op = bundleoperation(repo, lambda: tr)
1d2b68952950 bundle: inline applybundle1() Martin von Zweigbergk <martinvonz@google.com> parents: 33826 diff changeset	347 _processchangegroup(op, unbundler, tr, source, url, **kwargs)
1d2b68952950 bundle: inline applybundle1() Martin von Zweigbergk <martinvonz@google.com> parents: 33826 diff changeset	348 return op
27430 28a6c2d72097 bundle2: introduce an "applybundle" function Pierre-Yves David <pierre-yves.david@fb.com> parents: 27338 diff changeset	349
25056 df0ce98c882f bundle2: also save output when error happens during part processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25054 diff changeset	350 def processbundle(repo, unbundler, transactiongetter=None, op=None):
20889 deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	351 """This function process a bundle, apply effect to/from a repo
deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	352
20947 c33d7bf53812 bundle2: feed a unbundle20 to the `processbundle` function Pierre-Yves David <pierre-yves.david@fb.com> parents: 20892 diff changeset	353 It iterates over each part then searches for and uses the proper handling
c33d7bf53812 bundle2: feed a unbundle20 to the `processbundle` function Pierre-Yves David <pierre-yves.david@fb.com> parents: 20892 diff changeset	354 code to process the part. Parts are processed in order.
20889 deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	355
20891 1c6cd23fc221 bundle2: add some distinction between mandatory and advisory part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20890 diff changeset	356 Unknown Mandatory part will abort the process.
25056 df0ce98c882f bundle2: also save output when error happens during part processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25054 diff changeset	357
df0ce98c882f bundle2: also save output when error happens during part processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25054 diff changeset	358 It is temporarily possible to provide a prebuilt bundleoperation to the
df0ce98c882f bundle2: also save output when error happens during part processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25054 diff changeset	359 function. This is used to ensure output is properly propagated in case of
df0ce98c882f bundle2: also save output when error happens during part processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25054 diff changeset	360 an error during the unbundling. This output capturing part will likely be
df0ce98c882f bundle2: also save output when error happens during part processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25054 diff changeset	361 reworked and this ability will probably go away in the process.
20889 deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	362 """
25056 df0ce98c882f bundle2: also save output when error happens during part processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25054 diff changeset	363 if op is None:
df0ce98c882f bundle2: also save output when error happens during part processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25054 diff changeset	364 if transactiongetter is None:
df0ce98c882f bundle2: also save output when error happens during part processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25054 diff changeset	365 transactiongetter = _notransaction
df0ce98c882f bundle2: also save output when error happens during part processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25054 diff changeset	366 op = bundleoperation(repo, transactiongetter)
20889 deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	367 # todo:
deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	368 # - replace this is a init function soon.
deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	369 # - exception catching
deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	370 unbundler.params
25815 53ca47ef645d bundle2: add generic debug output regarding processed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 25808 diff changeset	371 if repo.ui.debugflag:
53ca47ef645d bundle2: add generic debug output regarding processed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 25808 diff changeset	372 msg = ['bundle2-input-bundle:']
53ca47ef645d bundle2: add generic debug output regarding processed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 25808 diff changeset	373 if unbundler.params:
33911 126eae7dae74 bundle2: add debug info about the number of stream params Siddharth Agarwal <sid0@fb.com> parents: 33910 diff changeset	374 msg.append(' %i params' % len(unbundler.params))
34553 119e1c6be1ce bundle2: fix transaction availability detection Boris Feld <boris.feld@octobus.net> parents: 34494 diff changeset	375 if op._gettransaction is None or op._gettransaction is _notransaction:
25815 53ca47ef645d bundle2: add generic debug output regarding processed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 25808 diff changeset	376 msg.append(' no-transaction')
53ca47ef645d bundle2: add generic debug output regarding processed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 25808 diff changeset	377 else:
53ca47ef645d bundle2: add generic debug output regarding processed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 25808 diff changeset	378 msg.append(' with-transaction')
53ca47ef645d bundle2: add generic debug output regarding processed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 25808 diff changeset	379 msg.append('\n')
53ca47ef645d bundle2: add generic debug output regarding processed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 25808 diff changeset	380 repo.ui.debug(''.join(msg))
25816 0cdd1c811705 bundle2: add generic debug output at the end of bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25815 diff changeset	381 iterparts = enumerate(unbundler.iterparts())
21019 3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	382 part = None
25816 0cdd1c811705 bundle2: add generic debug output at the end of bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25815 diff changeset	383 nbpart = 0
20892 6fe95448596d bundle2: read the whole bundle from stream on abort Pierre-Yves David <pierre-yves.david@fb.com> parents: 20891 diff changeset	384 try:
25816 0cdd1c811705 bundle2: add generic debug output at the end of bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25815 diff changeset	385 for nbpart, part in iterparts:
23008 d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	386 _processpart(op, part)
30617 9a9629b9416c bundle2: fail faster when interrupted Gregory Szorc <gregory.szorc@gmail.com> parents: 30354 diff changeset	387 except Exception as exc:
32803 ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	388 # Any exceptions seeking to the end of the bundle at this point are
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	389 # almost certainly related to the underlying stream being bad.
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	390 # And, chances are that the exception we're handling is related to
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	391 # getting in that bad state. So, we swallow the seeking error and
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	392 # re-raise the original error.
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	393 seekerror = False
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	394 try:
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	395 for nbpart, part in iterparts:
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	396 # consume the bundle content
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	397 part.seek(0, 2)
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	398 except Exception:
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	399 seekerror = True
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	400
21176 70fcb0a71445 bundle2: decorate exception raised during bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 21146 diff changeset	401 # Small hack to let caller code distinguish exceptions from bundle2
23139 e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues Mads Kiilerich <madski@unity3d.com> parents: 23067 diff changeset	402 # processing from processing the old format. This is mostly
21176 70fcb0a71445 bundle2: decorate exception raised during bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 21146 diff changeset	403 # needed to handle different return codes to unbundle according to the
70fcb0a71445 bundle2: decorate exception raised during bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 21146 diff changeset	404 # type of bundle. We should probably clean up or drop this return code
70fcb0a71445 bundle2: decorate exception raised during bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 21146 diff changeset	405 # craziness in a future version.
70fcb0a71445 bundle2: decorate exception raised during bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 21146 diff changeset	406 exc.duringunbundle2 = True
24974 f9aa4cb8f2dd bundle2: store the salvaged output on the exception object Pierre-Yves David <pierre-yves.david@fb.com> parents: 24973 diff changeset	407 salvaged = []
26028 219b8ab31220 bundle2: also capture reply capability on failure Pierre-Yves David <pierre-yves.david@fb.com> parents: 26027 diff changeset	408 replycaps = None
24974 f9aa4cb8f2dd bundle2: store the salvaged output on the exception object Pierre-Yves David <pierre-yves.david@fb.com> parents: 24973 diff changeset	409 if op.reply is not None:
f9aa4cb8f2dd bundle2: store the salvaged output on the exception object Pierre-Yves David <pierre-yves.david@fb.com> parents: 24973 diff changeset	410 salvaged = op.reply.salvageoutput()
26028 219b8ab31220 bundle2: also capture reply capability on failure Pierre-Yves David <pierre-yves.david@fb.com> parents: 26027 diff changeset	411 replycaps = op.reply.capabilities
219b8ab31220 bundle2: also capture reply capability on failure Pierre-Yves David <pierre-yves.david@fb.com> parents: 26027 diff changeset	412 exc._replycaps = replycaps
24974 f9aa4cb8f2dd bundle2: store the salvaged output on the exception object Pierre-Yves David <pierre-yves.david@fb.com> parents: 24973 diff changeset	413 exc._bundle2salvagedoutput = salvaged
32803 ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	414
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	415 # Re-raising from a variable loses the original stack. So only use
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	416 # that form if we need to.
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	417 if seekerror:
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	418 raise exc
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	419 else:
ad41739c6b2b bundle2: ignore errors seeking a bundle after an exception (issue4784) Gregory Szorc <gregory.szorc@gmail.com> parents: 32668 diff changeset	420 raise
25816 0cdd1c811705 bundle2: add generic debug output at the end of bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25815 diff changeset	421 finally:
0cdd1c811705 bundle2: add generic debug output at the end of bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25815 diff changeset	422 repo.ui.debug('bundle2-input-bundle: %i parts total\n' % nbpart)
0cdd1c811705 bundle2: add generic debug output at the end of bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25815 diff changeset	423
20949 571f2903ff1e bundle2: record processing results in the bundleoperation object Pierre-Yves David <pierre-yves.david@fb.com> parents: 20948 diff changeset	424 return op
20889 deed5edb72de bundle2: first version of a bundle processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 20887 diff changeset	425
33821 f0efd2bffe1e bundle: extract _processchangegroup() method Martin von Zweigbergk <martinvonz@google.com> parents: 33820 diff changeset	426 def _processchangegroup(op, cg, tr, source, url, **kwargs):
34244 bb72031f0ea8 changegroup: stop returning and recording added nodes in 'cg.apply' Boris Feld <boris.feld@octobus.net> parents: 34243 diff changeset	427 ret = cg.apply(op.repo, tr, source, url, **kwargs)
33821 f0efd2bffe1e bundle: extract _processchangegroup() method Martin von Zweigbergk <martinvonz@google.com> parents: 33820 diff changeset	428 op.records.add('changegroup', {
f0efd2bffe1e bundle: extract _processchangegroup() method Martin von Zweigbergk <martinvonz@google.com> parents: 33820 diff changeset	429 'return': ret,
f0efd2bffe1e bundle: extract _processchangegroup() method Martin von Zweigbergk <martinvonz@google.com> parents: 33820 diff changeset	430 })
f0efd2bffe1e bundle: extract _processchangegroup() method Martin von Zweigbergk <martinvonz@google.com> parents: 33820 diff changeset	431 return ret
f0efd2bffe1e bundle: extract _processchangegroup() method Martin von Zweigbergk <martinvonz@google.com> parents: 33820 diff changeset	432
23008 d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	433 def _processpart(op, part):
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	434 """process a single part from a bundle
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	435
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	436 The part is guaranteed to have been fully consumed when the function exits
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	437 (even if an exception is raised)."""
25817 5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	438 status = 'unknown' # used by debug output
30617 9a9629b9416c bundle2: fail faster when interrupted Gregory Szorc <gregory.szorc@gmail.com> parents: 30354 diff changeset	439 hardabort = False
23008 d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	440 try:
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	441 try:
23598 112f9c73a0e5 bundle2._processpart: forcing lower-case compare is no longer necessary Eric Sumner <ericsumner@fb.com> parents: 23597 diff changeset	442 handler = parthandlermapping.get(part.type)
23008 d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	443 if handler is None:
25817 5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	444 status = 'unsupported-type'
27019 cff70549a959 bundle2: rename error exception class for unsupported feature Pierre-Yves David <pierre-yves.david@fb.com> parents: 26887 diff changeset	445 raise error.BundleUnknownFeatureError(parttype=part.type)
25804 697d8953b04d bundle2: handle new line in 'indebug' function Pierre-Yves David <pierre-yves.david@fb.com> parents: 25803 diff changeset	446 indebug(op.ui, 'found a handler for part %r' % part.type)
23008 d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	447 unknownparams = part.mandatorykeys - handler.params
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	448 if unknownparams:
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	449 unknownparams = list(unknownparams)
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	450 unknownparams.sort()
25817 5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	451 status = 'unsupported-params (%s)' % unknownparams
27019 cff70549a959 bundle2: rename error exception class for unsupported feature Pierre-Yves David <pierre-yves.david@fb.com> parents: 26887 diff changeset	452 raise error.BundleUnknownFeatureError(parttype=part.type,
cff70549a959 bundle2: rename error exception class for unsupported feature Pierre-Yves David <pierre-yves.david@fb.com> parents: 26887 diff changeset	453 params=unknownparams)
25817 5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	454 status = 'supported'
27019 cff70549a959 bundle2: rename error exception class for unsupported feature Pierre-Yves David <pierre-yves.david@fb.com> parents: 26887 diff changeset	455 except error.BundleUnknownFeatureError as exc:
23597 94b25d71dd0f bundle2.unbundlepart: decouple mandatory from parttype Eric Sumner <ericsumner@fb.com> parents: 23439 diff changeset	456 if part.mandatory: # mandatory parts
23008 d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	457 raise
25804 697d8953b04d bundle2: handle new line in 'indebug' function Pierre-Yves David <pierre-yves.david@fb.com> parents: 25803 diff changeset	458 indebug(op.ui, 'ignoring unsupported advisory part %s' % exc)
23008 d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	459 return # skip to part processing
25817 5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	460 finally:
5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	461 if op.ui.debugflag:
5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	462 msg = ['bundle2-input-part: "%s"' % part.type]
5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	463 if not part.mandatory:
5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	464 msg.append(' (advisory)')
5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	465 nbmp = len(part.mandatorykeys)
5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	466 nbap = len(part.params) - nbmp
5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	467 if nbmp or nbap:
5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	468 msg.append(' (params:')
5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	469 if nbmp:
5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	470 msg.append(' %i mandatory' % nbmp)
5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	471 if nbap:
5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	472 msg.append(' %i advisory' % nbmp)
5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	473 msg.append(')')
5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	474 msg.append(' %s\n' % status)
5bfffdbdc907 bundle2: add generic debug output regarding processed part Pierre-Yves David <pierre-yves.david@fb.com> parents: 25816 diff changeset	475 op.ui.debug(''.join(msg))
23008 d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	476
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	477 # handler is called outside the above try block so that we don't
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	478 # risk catching KeyErrors from anything other than the
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	479 # parthandlermapping lookup (any KeyError raised by handler()
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	480 # itself represents a defect of a different variety).
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	481 output = None
25135 e530cde6d115 bundle2: disable ouput capture unless we use http (issue4613 issue4615) Pierre-Yves David <pierre-yves.david@fb.com> parents: 25056 diff changeset	482 if op.captureoutput and op.reply is not None:
25054 aff2aca3420e bundle2: also capture hook output during processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 25051 diff changeset	483 op.ui.pushbuffer(error=True, subproc=True)
23008 d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	484 output = ''
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	485 try:
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	486 handler(op, part)
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	487 finally:
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	488 if output is not None:
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	489 output = op.ui.popbuffer()
24922 a2ef1dc3b59b bundle2: flush output in a part in all cases Pierre-Yves David <pierre-yves.david@fb.com> parents: 24920 diff changeset	490 if output:
a2ef1dc3b59b bundle2: flush output in a part in all cases Pierre-Yves David <pierre-yves.david@fb.com> parents: 24920 diff changeset	491 outpart = op.reply.newpart('output', data=output,
a2ef1dc3b59b bundle2: flush output in a part in all cases Pierre-Yves David <pierre-yves.david@fb.com> parents: 24920 diff changeset	492 mandatory=False)
34458 24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes Augie Fackler <augie@google.com> parents: 34457 diff changeset	493 outpart.addparam(
24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes Augie Fackler <augie@google.com> parents: 34457 diff changeset	494 'in-reply-to', pycompat.bytestr(part.id), mandatory=False)
30617 9a9629b9416c bundle2: fail faster when interrupted Gregory Szorc <gregory.szorc@gmail.com> parents: 30354 diff changeset	495 # If exiting or interrupted, do not attempt to seek the stream in the
9a9629b9416c bundle2: fail faster when interrupted Gregory Szorc <gregory.szorc@gmail.com> parents: 30354 diff changeset	496 # finally block below. This makes abort faster.
9a9629b9416c bundle2: fail faster when interrupted Gregory Szorc <gregory.szorc@gmail.com> parents: 30354 diff changeset	497 except (SystemExit, KeyboardInterrupt):
9a9629b9416c bundle2: fail faster when interrupted Gregory Szorc <gregory.szorc@gmail.com> parents: 30354 diff changeset	498 hardabort = True
9a9629b9416c bundle2: fail faster when interrupted Gregory Szorc <gregory.szorc@gmail.com> parents: 30354 diff changeset	499 raise
23008 d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	500 finally:
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	501 # consume the part content to not corrupt the stream.
30617 9a9629b9416c bundle2: fail faster when interrupted Gregory Szorc <gregory.szorc@gmail.com> parents: 30354 diff changeset	502 if not hardabort:
9a9629b9416c bundle2: fail faster when interrupted Gregory Szorc <gregory.szorc@gmail.com> parents: 30354 diff changeset	503 part.seek(0, 2)
23008 d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	504
d3137827016a bundle2: extract processing of part into its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 23001 diff changeset	505
21138 f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	506 def decodecaps(blob):
23139 e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues Mads Kiilerich <madski@unity3d.com> parents: 23067 diff changeset	507 """decode a bundle2 caps bytes blob into a dictionary
21138 f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	508
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	509 The blob is a list of capabilities (one per line)
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	510 Capabilities may have values using a line of the form::
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	511
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	512 capability=value1,value2,value3
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	513
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	514 The values are always a list."""
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	515 caps = {}
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	516 for line in blob.splitlines():
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	517 if not line:
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	518 continue
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	519 if '=' not in line:
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	520 key, vals = line, ()
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	521 else:
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	522 key, vals = line.split('=', 1)
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	523 vals = vals.split(',')
29633 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 29422 diff changeset	524 key = urlreq.unquote(key)
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 29422 diff changeset	525 vals = [urlreq.unquote(v) for v in vals]
21138 f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	526 caps[key] = vals
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	527 return caps
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	528
21139 2b8c82f7f11d bundle2: capabilities encoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21138 diff changeset	529 def encodecaps(caps):
2b8c82f7f11d bundle2: capabilities encoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21138 diff changeset	530 """encode a bundle2 caps dictionary into a bytes blob"""
2b8c82f7f11d bundle2: capabilities encoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21138 diff changeset	531 chunks = []
2b8c82f7f11d bundle2: capabilities encoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21138 diff changeset	532 for ca in sorted(caps):
2b8c82f7f11d bundle2: capabilities encoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21138 diff changeset	533 vals = caps[ca]
29633 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 29422 diff changeset	534 ca = urlreq.quote(ca)
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 29422 diff changeset	535 vals = [urlreq.quote(v) for v in vals]
21139 2b8c82f7f11d bundle2: capabilities encoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21138 diff changeset	536 if vals:
2b8c82f7f11d bundle2: capabilities encoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21138 diff changeset	537 ca = "%s=%s" % (ca, ','.join(vals))
2b8c82f7f11d bundle2: capabilities encoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21138 diff changeset	538 chunks.append(ca)
2b8c82f7f11d bundle2: capabilities encoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21138 diff changeset	539 return '\n'.join(chunks)
2b8c82f7f11d bundle2: capabilities encoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21138 diff changeset	540
29407 ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	541 bundletypes = {
31130 f81002f736d7 bundle2: use new compression engine API for compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30956 diff changeset	542 "": ("", 'UN'), # only when using unbundle on ssh and old http servers
29407 ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	543 # since the unification ssh accepts a header but there
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	544 # is no capability signaling it.
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	545 "HG20": (), # special-cased below
31130 f81002f736d7 bundle2: use new compression engine API for compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30956 diff changeset	546 "HG10UN": ("HG10UN", 'UN'),
29407 ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	547 "HG10BZ": ("HG10", 'BZ'),
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	548 "HG10GZ": ("HG10GZ", 'GZ'),
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	549 }
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	550
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	551 # hgweb uses this list to communicate its preferred type
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	552 bundlepriority = ['HG10GZ', 'HG10BZ', 'HG10UN']
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	553
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	554 class bundle20(object):
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	555 """represent an outgoing bundle2 container
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	556
21599 57cd844d7a5b bundle2: have ``newpart`` automatically add the part to the bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 21598 diff changeset	557 Use the `addparam` method to add stream level parameter. and `newpart` to
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	558 populate it. Then call `getchunks` to retrieve all the binary chunks of
21024 7731a2281cf0 spelling: fixes from spell checker Mads Kiilerich <madski@unity3d.com> parents: 21020 diff changeset	559 data that compose the bundle2 container."""
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	560
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	561 _magicstring = 'HG20'
24773 685639f9430d bundle20: move magic string into the class Pierre-Yves David <pierre-yves.david@fb.com> parents: 24083 diff changeset	562
21134 2f8c4fa237f5 bundle2: adds a capabilities attribute on bundler20 Pierre-Yves David <pierre-yves.david@fb.com> parents: 21133 diff changeset	563 def __init__(self, ui, capabilities=()):
20842 938718d72624 bundle2: print debug information during bundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 20814 diff changeset	564 self.ui = ui
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	565 self._params = []
9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	566 self._parts = []
21136 b6fd496e5c72 bundle2: support for capabilities with values Pierre-Yves David <pierre-yves.david@fb.com> parents: 21135 diff changeset	567 self.capabilities = dict(capabilities)
31130 f81002f736d7 bundle2: use new compression engine API for compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30956 diff changeset	568 self._compengine = util.compengines.forbundletype('UN')
31536 511a4bf52754 bundle2: allow compression options to be passed to compressor Gregory Szorc <gregory.szorc@gmail.com> parents: 31218 diff changeset	569 self._compopts = None
27030 795f02a24b87 bundle2: allow compressed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 27022 diff changeset	570
31536 511a4bf52754 bundle2: allow compression options to be passed to compressor Gregory Szorc <gregory.szorc@gmail.com> parents: 31218 diff changeset	571 def setcompression(self, alg, compopts=None):
27030 795f02a24b87 bundle2: allow compressed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 27022 diff changeset	572 """setup core part compression to <alg>"""
31218 71b368e3b590 bundle2: equate 'UN' with no compression Gregory Szorc <gregory.szorc@gmail.com> parents: 31136 diff changeset	573 if alg in (None, 'UN'):
27030 795f02a24b87 bundle2: allow compressed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 27022 diff changeset	574 return
31694 aa25989b0658 bundle2: fix assertion that 'compression' hasn't been set Siddharth Agarwal <sid0@fb.com> parents: 31536 diff changeset	575 assert not any(n.lower() == 'compression' for n, v in self._params)
27030 795f02a24b87 bundle2: allow compressed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 27022 diff changeset	576 self.addparam('Compression', alg)
31130 f81002f736d7 bundle2: use new compression engine API for compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30956 diff changeset	577 self._compengine = util.compengines.forbundletype(alg)
31536 511a4bf52754 bundle2: allow compression options to be passed to compressor Gregory Szorc <gregory.szorc@gmail.com> parents: 31218 diff changeset	578 self._compopts = compopts
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	579
21900 b8bd97085ec9 bundle2: add a ``bundle20.nbparts`` property Pierre-Yves David <pierre-yves.david@fb.com> parents: 21660 diff changeset	580 @property
b8bd97085ec9 bundle2: add a ``bundle20.nbparts`` property Pierre-Yves David <pierre-yves.david@fb.com> parents: 21660 diff changeset	581 def nbparts(self):
b8bd97085ec9 bundle2: add a ``bundle20.nbparts`` property Pierre-Yves David <pierre-yves.david@fb.com> parents: 21660 diff changeset	582 """total number of parts added to the bundler"""
b8bd97085ec9 bundle2: add a ``bundle20.nbparts`` property Pierre-Yves David <pierre-yves.david@fb.com> parents: 21660 diff changeset	583 return len(self._parts)
b8bd97085ec9 bundle2: add a ``bundle20.nbparts`` property Pierre-Yves David <pierre-yves.david@fb.com> parents: 21660 diff changeset	584
21597 1daad9dcdba2 bundle2: small doc update on the bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21186 diff changeset	585 # methods used to defines the bundle2 content
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	586 def addparam(self, name, value=None):
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	587 """add a stream level parameter"""
20813 8c74b3ce5b70 bundle2: refuse empty parameter name Pierre-Yves David <pierre-yves.david@fb.com> parents: 20812 diff changeset	588 if not name:
8c74b3ce5b70 bundle2: refuse empty parameter name Pierre-Yves David <pierre-yves.david@fb.com> parents: 20812 diff changeset	589 raise ValueError('empty parameter name')
34665 8d75b9effa07 bundle2: fixed usage of an attribute that was removed in py3k Alex Gaynor <alex.gaynor@gmail.com> parents: 34618 diff changeset	590 if name[0] not in pycompat.bytestr(string.ascii_letters):
20814 8532f5e1b9df bundle2: force the first char of parameter to be an letter. Pierre-Yves David <pierre-yves.david@fb.com> parents: 20813 diff changeset	591 raise ValueError('non letter first character: %r' % name)
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	592 self._params.append((name, value))
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	593
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	594 def addpart(self, part):
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	595 """add a new part to the bundle2 container
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	596
21024 7731a2281cf0 spelling: fixes from spell checker Mads Kiilerich <madski@unity3d.com> parents: 21020 diff changeset	597 Parts contains the actual applicative payload."""
20995 e995d104c87f bundle2: add an integer id to part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20952 diff changeset	598 assert part.id is None
e995d104c87f bundle2: add an integer id to part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20952 diff changeset	599 part.id = len(self._parts) # very cheap counter
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	600 self._parts.append(part)
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	601
21598 1b0dbb91de5b bundle2: add a ``newpart`` method to ``bundle20`` Pierre-Yves David <pierre-yves.david@fb.com> parents: 21597 diff changeset	602 def newpart(self, typeid, args, *kwargs):
21602 cc33ae50bab3 bundle2: warn about error during initialization in ``newpart`` docstring Pierre-Yves David <pierre-yves.david@fb.com> parents: 21601 diff changeset	603 """create a new part and add it to the containers
cc33ae50bab3 bundle2: warn about error during initialization in ``newpart`` docstring Pierre-Yves David <pierre-yves.david@fb.com> parents: 21601 diff changeset	604
cc33ae50bab3 bundle2: warn about error during initialization in ``newpart`` docstring Pierre-Yves David <pierre-yves.david@fb.com> parents: 21601 diff changeset	605 As the part is directly added to the containers. For now, this means
cc33ae50bab3 bundle2: warn about error during initialization in ``newpart`` docstring Pierre-Yves David <pierre-yves.david@fb.com> parents: 21601 diff changeset	606 that any failure to properly initialize the part after calling
cc33ae50bab3 bundle2: warn about error during initialization in ``newpart`` docstring Pierre-Yves David <pierre-yves.david@fb.com> parents: 21601 diff changeset	607 ``newpart`` should result in a failure of the whole bundling process.
cc33ae50bab3 bundle2: warn about error during initialization in ``newpart`` docstring Pierre-Yves David <pierre-yves.david@fb.com> parents: 21601 diff changeset	608
cc33ae50bab3 bundle2: warn about error during initialization in ``newpart`` docstring Pierre-Yves David <pierre-yves.david@fb.com> parents: 21601 diff changeset	609 You can still fall back to manually create and add if you need better
cc33ae50bab3 bundle2: warn about error during initialization in ``newpart`` docstring Pierre-Yves David <pierre-yves.david@fb.com> parents: 21601 diff changeset	610 control."""
21598 1b0dbb91de5b bundle2: add a ``newpart`` method to ``bundle20`` Pierre-Yves David <pierre-yves.david@fb.com> parents: 21597 diff changeset	611 part = bundlepart(typeid, args, *kwargs)
21599 57cd844d7a5b bundle2: have ``newpart`` automatically add the part to the bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 21598 diff changeset	612 self.addpart(part)
21598 1b0dbb91de5b bundle2: add a ``newpart`` method to ``bundle20`` Pierre-Yves David <pierre-yves.david@fb.com> parents: 21597 diff changeset	613 return part
1b0dbb91de5b bundle2: add a ``newpart`` method to ``bundle20`` Pierre-Yves David <pierre-yves.david@fb.com> parents: 21597 diff changeset	614
21597 1daad9dcdba2 bundle2: small doc update on the bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21186 diff changeset	615 # methods used to generate the bundle2 stream
20801 9c5183cb9bca bundle2: very first version of a bundle2 bundler Pierre-Yves David <pierre-yves.david@fb.com> parents: diff changeset	616 def getchunks(self):
25806 1f7c0403f8be bundle2: add generic debug output regarding generated bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 25805 diff changeset	617 if self.ui.debugflag:
1f7c0403f8be bundle2: add generic debug output regarding generated bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 25805 diff changeset	618 msg = ['bundle2-output-bundle: "%s",' % self._magicstring]
1f7c0403f8be bundle2: add generic debug output regarding generated bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 25805 diff changeset	619 if self._params:
1f7c0403f8be bundle2: add generic debug output regarding generated bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 25805 diff changeset	620 msg.append(' (%i params)' % len(self._params))
1f7c0403f8be bundle2: add generic debug output regarding generated bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 25805 diff changeset	621 msg.append(' %i parts total\n' % len(self._parts))
1f7c0403f8be bundle2: add generic debug output regarding generated bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 25805 diff changeset	622 self.ui.debug(''.join(msg))
25799 e28245b0da56 bundle2: handle new line in 'outdebug' function Pierre-Yves David <pierre-yves.david@fb.com> parents: 25798 diff changeset	623 outdebug(self.ui, 'start emission of %s stream' % self._magicstring)
24773 685639f9430d bundle20: move magic string into the class Pierre-Yves David <pierre-yves.david@fb.com> parents: 24083 diff changeset	624 yield self._magicstring
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	625 param = self._paramchunk()
25799 e28245b0da56 bundle2: handle new line in 'outdebug' function Pierre-Yves David <pierre-yves.david@fb.com> parents: 25798 diff changeset	626 outdebug(self.ui, 'bundle parameter: %s' % param)
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	627 yield _pack(_fstreamparamsize, len(param))
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	628 if param:
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	629 yield param
31536 511a4bf52754 bundle2: allow compression options to be passed to compressor Gregory Szorc <gregory.szorc@gmail.com> parents: 31218 diff changeset	630 for chunk in self._compengine.compressstream(self._getcorechunk(),
511a4bf52754 bundle2: allow compression options to be passed to compressor Gregory Szorc <gregory.szorc@gmail.com> parents: 31218 diff changeset	631 self._compopts):
31136 5925bda42dbd bundle2: use compressstream compression engine API Gregory Szorc <gregory.szorc@gmail.com> parents: 31132 diff changeset	632 yield chunk
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	633
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	634 def _paramchunk(self):
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	635 """return a encoded version of all stream parameters"""
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	636 blocks = []
20809 b93bb639451a bundle2: support for bundling parameter value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20808 diff changeset	637 for par, value in self._params:
29633 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 29422 diff changeset	638 par = urlreq.quote(par)
20809 b93bb639451a bundle2: support for bundling parameter value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20808 diff changeset	639 if value is not None:
29633 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 29422 diff changeset	640 value = urlreq.quote(value)
20809 b93bb639451a bundle2: support for bundling parameter value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20808 diff changeset	641 par = '%s=%s' % (par, value)
b93bb639451a bundle2: support for bundling parameter value Pierre-Yves David <pierre-yves.david@fb.com> parents: 20808 diff changeset	642 blocks.append(par)
20804 db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	643 return ' '.join(blocks)
db9d3991d2c6 bundle2: support bundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20803 diff changeset	644
27022 d90c30801cdf bundle20: extract core payload generation in its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 27021 diff changeset	645 def _getcorechunk(self):
d90c30801cdf bundle20: extract core payload generation in its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 27021 diff changeset	646 """yield chunk for the core part of the bundle
d90c30801cdf bundle20: extract core payload generation in its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 27021 diff changeset	647
d90c30801cdf bundle20: extract core payload generation in its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 27021 diff changeset	648 (all but headers and parameters)"""
d90c30801cdf bundle20: extract core payload generation in its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 27021 diff changeset	649 outdebug(self.ui, 'start of parts')
d90c30801cdf bundle20: extract core payload generation in its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 27021 diff changeset	650 for part in self._parts:
d90c30801cdf bundle20: extract core payload generation in its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 27021 diff changeset	651 outdebug(self.ui, 'bundle part: "%s"' % part.type)
d90c30801cdf bundle20: extract core payload generation in its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 27021 diff changeset	652 for chunk in part.getchunks(ui=self.ui):
d90c30801cdf bundle20: extract core payload generation in its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 27021 diff changeset	653 yield chunk
d90c30801cdf bundle20: extract core payload generation in its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 27021 diff changeset	654 outdebug(self.ui, 'end of bundle')
d90c30801cdf bundle20: extract core payload generation in its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 27021 diff changeset	655 yield _pack(_fpartheadersize, 0)
d90c30801cdf bundle20: extract core payload generation in its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 27021 diff changeset	656
d90c30801cdf bundle20: extract core payload generation in its own function Pierre-Yves David <pierre-yves.david@fb.com> parents: 27021 diff changeset	657
24973 21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20 Pierre-Yves David <pierre-yves.david@fb.com> parents: 24972 diff changeset	658 def salvageoutput(self):
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20 Pierre-Yves David <pierre-yves.david@fb.com> parents: 24972 diff changeset	659 """return a list with a copy of all output parts in the bundle
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20 Pierre-Yves David <pierre-yves.david@fb.com> parents: 24972 diff changeset	660
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20 Pierre-Yves David <pierre-yves.david@fb.com> parents: 24972 diff changeset	661 This is meant to be used during error handling to make sure we preserve
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20 Pierre-Yves David <pierre-yves.david@fb.com> parents: 24972 diff changeset	662 server output"""
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20 Pierre-Yves David <pierre-yves.david@fb.com> parents: 24972 diff changeset	663 salvaged = []
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20 Pierre-Yves David <pierre-yves.david@fb.com> parents: 24972 diff changeset	664 for part in self._parts:
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20 Pierre-Yves David <pierre-yves.david@fb.com> parents: 24972 diff changeset	665 if part.type.startswith('output'):
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20 Pierre-Yves David <pierre-yves.david@fb.com> parents: 24972 diff changeset	666 salvaged.append(part.copy())
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20 Pierre-Yves David <pierre-yves.david@fb.com> parents: 24972 diff changeset	667 return salvaged
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20 Pierre-Yves David <pierre-yves.david@fb.com> parents: 24972 diff changeset	668
21f2e8f48ae1 bundle2: add a 'salvageoutput' method on bundle20 Pierre-Yves David <pierre-yves.david@fb.com> parents: 24972 diff changeset	669
21013 a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin Pierre-Yves David <pierre-yves.david@fb.com> parents: 21005 diff changeset	670 class unpackermixin(object):
a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin Pierre-Yves David <pierre-yves.david@fb.com> parents: 21005 diff changeset	671 """A mixin to extract bytes and struct data from a stream"""
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	672
21013 a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin Pierre-Yves David <pierre-yves.david@fb.com> parents: 21005 diff changeset	673 def __init__(self, fp):
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	674 self._fp = fp
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	675
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	676 def _unpack(self, format):
32641 9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32640 diff changeset	677 """unpack this struct format from the stream
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32640 diff changeset	678
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32640 diff changeset	679 This method is meant for internal usage by the bundle2 protocol only.
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32640 diff changeset	680 They directly manipulate the low level stream including bundle2 level
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32640 diff changeset	681 instruction.
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32640 diff changeset	682
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32640 diff changeset	683 Do not use it to implement higher-level logic or methods."""
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	684 data = self._readexact(struct.calcsize(format))
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	685 return _unpack(format, data)
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	686
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	687 def _readexact(self, size):
32641 9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32640 diff changeset	688 """read exactly <size> bytes from the stream
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32640 diff changeset	689
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32640 diff changeset	690 This method is meant for internal usage by the bundle2 protocol only.
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32640 diff changeset	691 They directly manipulate the low level stream including bundle2 level
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32640 diff changeset	692 instruction.
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32640 diff changeset	693
9bd9e9cb3f69 bundle2: clarify the docstring of unpackermixin methods Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32640 diff changeset	694 Do not use it to implement higher-level logic or methods."""
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	695 return changegroup.readexactly(self._fp, size)
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	696
26194 39f0064a3079 bundle2.getunbundler: rename "header" to "magicstring" Pierre-Yves David <pierre-yves.david@fb.com> parents: 26061 diff changeset	697 def getunbundler(ui, fp, magicstring=None):
39f0064a3079 bundle2.getunbundler: rename "header" to "magicstring" Pierre-Yves David <pierre-yves.david@fb.com> parents: 26061 diff changeset	698 """return a valid unbundler object for a given magicstring"""
39f0064a3079 bundle2.getunbundler: rename "header" to "magicstring" Pierre-Yves David <pierre-yves.david@fb.com> parents: 26061 diff changeset	699 if magicstring is None:
39f0064a3079 bundle2.getunbundler: rename "header" to "magicstring" Pierre-Yves David <pierre-yves.david@fb.com> parents: 26061 diff changeset	700 magicstring = changegroup.readexactly(fp, 4)
39f0064a3079 bundle2.getunbundler: rename "header" to "magicstring" Pierre-Yves David <pierre-yves.david@fb.com> parents: 26061 diff changeset	701 magic, version = magicstring[0:2], magicstring[2:4]
24787 5cac3accdaa1 unbundle20: allow generic dispatch between unbundlers Pierre-Yves David <pierre-yves.david@fb.com> parents: 24775 diff changeset	702 if magic != 'HG':
33910 0aae80d14fed bundle2: add some debugging information to the not-a-bundle error Siddharth Agarwal <sid0@fb.com> parents: 33827 diff changeset	703 ui.debug(
0aae80d14fed bundle2: add some debugging information to the not-a-bundle error Siddharth Agarwal <sid0@fb.com> parents: 33827 diff changeset	704 "error: invalid magic: %r (version %r), should be 'HG'\n"
0aae80d14fed bundle2: add some debugging information to the not-a-bundle error Siddharth Agarwal <sid0@fb.com> parents: 33827 diff changeset	705 % (magic, version))
27227 56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 27198 diff changeset	706 raise error.Abort(_('not a Mercurial bundle'))
24787 5cac3accdaa1 unbundle20: allow generic dispatch between unbundlers Pierre-Yves David <pierre-yves.david@fb.com> parents: 24775 diff changeset	707 unbundlerclass = formatmap.get(version)
5cac3accdaa1 unbundle20: allow generic dispatch between unbundlers Pierre-Yves David <pierre-yves.david@fb.com> parents: 24775 diff changeset	708 if unbundlerclass is None:
27227 56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 27198 diff changeset	709 raise error.Abort(_('unknown bundle version %s') % version)
24787 5cac3accdaa1 unbundle20: allow generic dispatch between unbundlers Pierre-Yves David <pierre-yves.david@fb.com> parents: 24775 diff changeset	710 unbundler = unbundlerclass(ui, fp)
26194 39f0064a3079 bundle2.getunbundler: rename "header" to "magicstring" Pierre-Yves David <pierre-yves.david@fb.com> parents: 26061 diff changeset	711 indebug(ui, 'start processing of %s stream' % magicstring)
24775 54e5c239c2d9 unbundle20: move header parsing into the 'getunbundler' function Pierre-Yves David <pierre-yves.david@fb.com> parents: 24774 diff changeset	712 return unbundler
24774 60fecc5b14a4 unbundle20: retrieve unbundler instances through a factory function Pierre-Yves David <pierre-yves.david@fb.com> parents: 24773 diff changeset	713
21013 a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin Pierre-Yves David <pierre-yves.david@fb.com> parents: 21005 diff changeset	714 class unbundle20(unpackermixin):
a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin Pierre-Yves David <pierre-yves.david@fb.com> parents: 21005 diff changeset	715 """interpret a bundle2 stream
a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin Pierre-Yves David <pierre-yves.david@fb.com> parents: 21005 diff changeset	716
21129 07bcbf326c8d bundle2: use an official iterparts method to unbundle parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 21066 diff changeset	717 This class is fed with a binary stream and yields parts through its
07bcbf326c8d bundle2: use an official iterparts method to unbundle parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 21066 diff changeset	718 `iterparts` methods."""
21013 a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin Pierre-Yves David <pierre-yves.david@fb.com> parents: 21005 diff changeset	719
27175 b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	720 _magicstring = 'HG20'
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	721
24775 54e5c239c2d9 unbundle20: move header parsing into the 'getunbundler' function Pierre-Yves David <pierre-yves.david@fb.com> parents: 24774 diff changeset	722 def __init__(self, ui, fp):
21066 5ecfe76d0d96 bundle2: make header reading optional Pierre-Yves David <pierre-yves.david@fb.com> parents: 21062 diff changeset	723 """If header is specified, we do not read it out of the stream."""
21013 a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin Pierre-Yves David <pierre-yves.david@fb.com> parents: 21005 diff changeset	724 self.ui = ui
31132 d045b4091197 bundle2: use compression engines API to obtain decompressor Gregory Szorc <gregory.szorc@gmail.com> parents: 31130 diff changeset	725 self._compengine = util.compengines.forbundletype('UN')
27442 42f705f2c02d bundle2: make unbundle.compressed return True when compressed Pierre-Yves David <pierre-yves.david@fb.com> parents: 27435 diff changeset	726 self._compressed = None
21013 a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin Pierre-Yves David <pierre-yves.david@fb.com> parents: 21005 diff changeset	727 super(unbundle20, self).__init__(fp)
a813caca89b3 bundle2: extract stream/unpack logic in an unpackermixin Pierre-Yves David <pierre-yves.david@fb.com> parents: 21005 diff changeset	728
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	729 @util.propertycache
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	730 def params(self):
21024 7731a2281cf0 spelling: fixes from spell checker Mads Kiilerich <madski@unity3d.com> parents: 21020 diff changeset	731 """dictionary of stream level parameters"""
25804 697d8953b04d bundle2: handle new line in 'indebug' function Pierre-Yves David <pierre-yves.david@fb.com> parents: 25803 diff changeset	732 indebug(self.ui, 'reading bundle2 stream parameters')
20805 c5aaeca0cfbf bundle2: support for unbundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20804 diff changeset	733 params = {}
c5aaeca0cfbf bundle2: support for unbundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20804 diff changeset	734 paramssize = self._unpack(_fstreamparamsize)[0]
23011 006a81d07e57 bundle2: detect and disallow a negative chunk size Pierre-Yves David <pierre-yves.david@fb.com> parents: 23010 diff changeset	735 if paramssize < 0:
006a81d07e57 bundle2: detect and disallow a negative chunk size Pierre-Yves David <pierre-yves.david@fb.com> parents: 23010 diff changeset	736 raise error.BundleValueError('negative bundle param size: %i'
006a81d07e57 bundle2: detect and disallow a negative chunk size Pierre-Yves David <pierre-yves.david@fb.com> parents: 23010 diff changeset	737 % paramssize)
20805 c5aaeca0cfbf bundle2: support for unbundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20804 diff changeset	738 if paramssize:
27174 d40029b4296e bundle2: split parameter retrieval and processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 27030 diff changeset	739 params = self._readexact(paramssize)
d40029b4296e bundle2: split parameter retrieval and processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 27030 diff changeset	740 params = self._processallparams(params)
20805 c5aaeca0cfbf bundle2: support for unbundling simple parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20804 diff changeset	741 return params
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	742
27174 d40029b4296e bundle2: split parameter retrieval and processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 27030 diff changeset	743 def _processallparams(self, paramsblock):
d40029b4296e bundle2: split parameter retrieval and processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 27030 diff changeset	744 """"""
30352 6215b5537ba5 bundle2: use a sorted dict for holding parameters Gregory Szorc <gregory.szorc@gmail.com> parents: 30044 diff changeset	745 params = util.sortdict()
27174 d40029b4296e bundle2: split parameter retrieval and processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 27030 diff changeset	746 for p in paramsblock.split(' '):
d40029b4296e bundle2: split parameter retrieval and processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 27030 diff changeset	747 p = p.split('=', 1)
29633 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 29422 diff changeset	748 p = [urlreq.unquote(i) for i in p]
27174 d40029b4296e bundle2: split parameter retrieval and processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 27030 diff changeset	749 if len(p) < 2:
d40029b4296e bundle2: split parameter retrieval and processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 27030 diff changeset	750 p.append(None)
d40029b4296e bundle2: split parameter retrieval and processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 27030 diff changeset	751 self._processparam(*p)
d40029b4296e bundle2: split parameter retrieval and processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 27030 diff changeset	752 params[p[0]] = p[1]
d40029b4296e bundle2: split parameter retrieval and processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 27030 diff changeset	753 return params
d40029b4296e bundle2: split parameter retrieval and processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 27030 diff changeset	754
d40029b4296e bundle2: split parameter retrieval and processing Pierre-Yves David <pierre-yves.david@fb.com> parents: 27030 diff changeset	755
20844 2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	756 def _processparam(self, name, value):
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	757 """process a parameter, applying its effect if needed
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	758
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	759 Parameter starting with a lower case letter are advisory and will be
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	760 ignored when unknown. Those starting with an upper case letter are
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	761 mandatory and will this function will raise a KeyError when unknown.
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	762
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	763 Note: no option are currently supported. Any input will be either
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	764 ignored or failing.
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	765 """
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	766 if not name:
2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	767 raise ValueError('empty parameter name')
34665 8d75b9effa07 bundle2: fixed usage of an attribute that was removed in py3k Alex Gaynor <alex.gaynor@gmail.com> parents: 34618 diff changeset	768 if name[0] not in pycompat.bytestr(string.ascii_letters):
20844 2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	769 raise ValueError('non letter first character: %r' % name)
27021 4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	770 try:
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	771 handler = b2streamparamsmap[name.lower()]
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	772 except KeyError:
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	773 if name[0].islower():
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	774 indebug(self.ui, "ignoring unknown parameter %r" % name)
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	775 else:
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	776 raise error.BundleUnknownFeatureError(params=(name,))
20844 2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	777 else:
27021 4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	778 handler(self, name, value)
20844 2631204d7305 bundle2: implement the mandatory/advisory logic for parameter Pierre-Yves David <pierre-yves.david@fb.com> parents: 20843 diff changeset	779
27175 b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	780 def _forwardchunks(self):
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	781 """utility to transfer a bundle2 as binary
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	782
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	783 This is made necessary by the fact the 'getbundle' command over 'ssh'
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	784 have no way to know then the reply end, relying on the bundle to be
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	785 interpreted to know its end. This is terrible and we are sorry, but we
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	786 needed to move forward to get general delta enabled.
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	787 """
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	788 yield self._magicstring
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	789 assert 'params' not in vars(self)
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	790 paramssize = self._unpack(_fstreamparamsize)[0]
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	791 if paramssize < 0:
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	792 raise error.BundleValueError('negative bundle param size: %i'
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	793 % paramssize)
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	794 yield _pack(_fstreamparamsize, paramssize)
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	795 if paramssize:
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	796 params = self._readexact(paramssize)
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	797 self._processallparams(params)
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	798 yield params
31132 d045b4091197 bundle2: use compression engines API to obtain decompressor Gregory Szorc <gregory.szorc@gmail.com> parents: 31130 diff changeset	799 assert self._compengine.bundletype == 'UN'
27175 b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	800 # From there, payload might need to be decompressed
31132 d045b4091197 bundle2: use compression engines API to obtain decompressor Gregory Szorc <gregory.szorc@gmail.com> parents: 31130 diff changeset	801 self._fp = self._compengine.decompressorreader(self._fp)
27175 b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	802 emptycount = 0
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	803 while emptycount < 2:
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	804 # so we can brainlessly loop
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	805 assert _fpartheadersize == _fpayloadsize
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	806 size = self._unpack(_fpartheadersize)[0]
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	807 yield _pack(_fpartheadersize, size)
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	808 if size:
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	809 emptycount = 0
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	810 else:
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	811 emptycount += 1
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	812 continue
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	813 if size == flaginterrupt:
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	814 continue
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	815 elif size < 0:
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	816 raise error.BundleValueError('negative chunk size: %i')
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	817 yield self._readexact(size)
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	818
b87e4638dabf bundle2: add a way to just forward the bundle2 stream to another user Pierre-Yves David <pierre-yves.david@fb.com> parents: 27174 diff changeset	819
21129 07bcbf326c8d bundle2: use an official iterparts method to unbundle parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 21066 diff changeset	820 def iterparts(self):
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	821 """yield all parts contained in the stream"""
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	822 # make sure param have been loaded
520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	823 self.params
27030 795f02a24b87 bundle2: allow compressed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 27022 diff changeset	824 # From there, payload need to be decompressed
31132 d045b4091197 bundle2: use compression engines API to obtain decompressor Gregory Szorc <gregory.szorc@gmail.com> parents: 31130 diff changeset	825 self._fp = self._compengine.decompressorreader(self._fp)
25804 697d8953b04d bundle2: handle new line in 'indebug' function Pierre-Yves David <pierre-yves.david@fb.com> parents: 25803 diff changeset	826 indebug(self.ui, 'start extraction of bundle2 parts')
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	827 headerblock = self._readpartheader()
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	828 while headerblock is not None:
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	829 part = unbundlepart(self.ui, headerblock, self._fp)
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	830 yield part
34671 891118dcd279 bundle2: seek part back during iteration Durham Goode <durham@fb.com> parents: 34669 diff changeset	831 # Seek to the end of the part to force it's consumption so the next
891118dcd279 bundle2: seek part back during iteration Durham Goode <durham@fb.com> parents: 34669 diff changeset	832 # part can be read. But then seek back to the beginning so the
891118dcd279 bundle2: seek part back during iteration Durham Goode <durham@fb.com> parents: 34669 diff changeset	833 # code consuming this generator has a part that starts at 0.
24060 13d88b7eb3a0 bundle2: seek in part iterator Eric Sumner <ericsumner@fb.com> parents: 24059 diff changeset	834 part.seek(0, 2)
34671 891118dcd279 bundle2: seek part back during iteration Durham Goode <durham@fb.com> parents: 34669 diff changeset	835 part.seek(0)
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	836 headerblock = self._readpartheader()
25804 697d8953b04d bundle2: handle new line in 'indebug' function Pierre-Yves David <pierre-yves.david@fb.com> parents: 25803 diff changeset	837 indebug(self.ui, 'end of bundle2 stream')
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	838
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	839 def _readpartheader(self):
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	840 """reads a part header size and return the bytes blob
20864 9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	841
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	842 returns None if empty"""
20864 9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	843 headersize = self._unpack(_fpartheadersize)[0]
23011 006a81d07e57 bundle2: detect and disallow a negative chunk size Pierre-Yves David <pierre-yves.david@fb.com> parents: 23010 diff changeset	844 if headersize < 0:
006a81d07e57 bundle2: detect and disallow a negative chunk size Pierre-Yves David <pierre-yves.david@fb.com> parents: 23010 diff changeset	845 raise error.BundleValueError('negative part header size: %i'
006a81d07e57 bundle2: detect and disallow a negative chunk size Pierre-Yves David <pierre-yves.david@fb.com> parents: 23010 diff changeset	846 % headersize)
25804 697d8953b04d bundle2: handle new line in 'indebug' function Pierre-Yves David <pierre-yves.david@fb.com> parents: 25803 diff changeset	847 indebug(self.ui, 'part header size: %i' % headersize)
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	848 if headersize:
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	849 return self._readexact(headersize)
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	850 return None
20864 9a75d2559cff bundle2: support unbundling empty part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20856 diff changeset	851
24083 184a2f6f40da bundle2.unbundle20: add compressed() method Eric Sumner <ericsumner@fb.com> parents: 24082 diff changeset	852 def compressed(self):
27442 42f705f2c02d bundle2: make unbundle.compressed return True when compressed Pierre-Yves David <pierre-yves.david@fb.com> parents: 27435 diff changeset	853 self.params # load params
42f705f2c02d bundle2: make unbundle.compressed return True when compressed Pierre-Yves David <pierre-yves.david@fb.com> parents: 27435 diff changeset	854 return self._compressed
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	855
32642 cd7aaf344d83 bundle2: move the 'close' method off the unpackermixin Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32641 diff changeset	856 def close(self):
cd7aaf344d83 bundle2: move the 'close' method off the unpackermixin Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32641 diff changeset	857 """close underlying file"""
cd7aaf344d83 bundle2: move the 'close' method off the unpackermixin Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32641 diff changeset	858 if util.safehasattr(self._fp, 'close'):
cd7aaf344d83 bundle2: move the 'close' method off the unpackermixin Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32641 diff changeset	859 return self._fp.close()
cd7aaf344d83 bundle2: move the 'close' method off the unpackermixin Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32641 diff changeset	860
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	861 formatmap = {'20': unbundle20}
24787 5cac3accdaa1 unbundle20: allow generic dispatch between unbundlers Pierre-Yves David <pierre-yves.david@fb.com> parents: 24775 diff changeset	862
27021 4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	863 b2streamparamsmap = {}
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	864
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	865 def b2streamparamhandler(name):
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	866 """register a handler for a stream level parameter"""
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	867 def decorator(func):
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	868 assert name not in formatmap
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	869 b2streamparamsmap[name] = func
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	870 return func
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	871 return decorator
4e7b0bf9f0b1 unbundle20: allow registering handlers for stream level parameters Pierre-Yves David <pierre-yves.david@fb.com> parents: 27019 diff changeset	872
27030 795f02a24b87 bundle2: allow compressed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 27022 diff changeset	873 @b2streamparamhandler('compression')
795f02a24b87 bundle2: allow compressed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 27022 diff changeset	874 def processcompression(unbundler, param, value):
795f02a24b87 bundle2: allow compressed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 27022 diff changeset	875 """read compression parameter and install payload decompression"""
31132 d045b4091197 bundle2: use compression engines API to obtain decompressor Gregory Szorc <gregory.szorc@gmail.com> parents: 31130 diff changeset	876 if value not in util.compengines.supportedbundletypes:
27030 795f02a24b87 bundle2: allow compressed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 27022 diff changeset	877 raise error.BundleUnknownFeatureError(params=(param,),
795f02a24b87 bundle2: allow compressed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 27022 diff changeset	878 values=(value,))
31132 d045b4091197 bundle2: use compression engines API to obtain decompressor Gregory Szorc <gregory.szorc@gmail.com> parents: 31130 diff changeset	879 unbundler._compengine = util.compengines.forbundletype(value)
27442 42f705f2c02d bundle2: make unbundle.compressed return True when compressed Pierre-Yves David <pierre-yves.david@fb.com> parents: 27435 diff changeset	880 if value is not None:
42f705f2c02d bundle2: make unbundle.compressed return True when compressed Pierre-Yves David <pierre-yves.david@fb.com> parents: 27435 diff changeset	881 unbundler._compressed = True
27030 795f02a24b87 bundle2: allow compressed bundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 27022 diff changeset	882
21005 3d38ebb586fe bundle2: rename part to bundlepart Pierre-Yves David <pierre-yves.david@fb.com> parents: 21004 diff changeset	883 class bundlepart(object):
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	884 """A bundle2 part contains application level payload
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	885
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	886 The part `type` is used to route the part to the application level
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	887 handler.
21604 c399bf961cb9 bundle2: the ability to set ``data`` attribute of the part is now official Pierre-Yves David <pierre-yves.david@fb.com> parents: 21603 diff changeset	888
c399bf961cb9 bundle2: the ability to set ``data`` attribute of the part is now official Pierre-Yves David <pierre-yves.david@fb.com> parents: 21603 diff changeset	889 The part payload is contained in ``part.data``. It could be raw bytes or a
21605 f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	890 generator of byte chunks.
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	891
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	892 You can add parameters to the part using the ``addparam`` method.
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	893 Parameters can be either mandatory (default) or advisory. Remote side
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	894 should be able to safely ignore the advisory ones.
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	895
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	896 Both data and parameters cannot be modified after the generation has begun.
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	897 """
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	898
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	899 def __init__(self, parttype, mandatoryparams=(), advisoryparams=(),
23602 4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API Eric Sumner <ericsumner@fb.com> parents: 23598 diff changeset	900 data='', mandatory=True):
23880 405eacbecc52 bundle2: enforce parttype as alphanumerical Pierre-Yves David <pierre-yves.david@fb.com> parents: 23602 diff changeset	901 validateparttype(parttype)
20995 e995d104c87f bundle2: add an integer id to part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20952 diff changeset	902 self.id = None
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	903 self.type = parttype
21604 c399bf961cb9 bundle2: the ability to set ``data`` attribute of the part is now official Pierre-Yves David <pierre-yves.david@fb.com> parents: 21603 diff changeset	904 self._data = data
21605 f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	905 self._mandatoryparams = list(mandatoryparams)
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	906 self._advisoryparams = list(advisoryparams)
21607 054fa5176fa7 bundle2: forbid duplicate parameter keys Pierre-Yves David <pierre-yves.david@fb.com> parents: 21606 diff changeset	907 # checking for duplicated entries
054fa5176fa7 bundle2: forbid duplicate parameter keys Pierre-Yves David <pierre-yves.david@fb.com> parents: 21606 diff changeset	908 self._seenparams = set()
054fa5176fa7 bundle2: forbid duplicate parameter keys Pierre-Yves David <pierre-yves.david@fb.com> parents: 21606 diff changeset	909 for pname, __ in self._mandatoryparams + self._advisoryparams:
054fa5176fa7 bundle2: forbid duplicate parameter keys Pierre-Yves David <pierre-yves.david@fb.com> parents: 21606 diff changeset	910 if pname in self._seenparams:
32426 4dbef666c6c9 bundle2: use ProgrammingError Jun Wu <quark@fb.com> parents: 32255 diff changeset	911 raise error.ProgrammingError('duplicated params: %s' % pname)
21607 054fa5176fa7 bundle2: forbid duplicate parameter keys Pierre-Yves David <pierre-yves.david@fb.com> parents: 21606 diff changeset	912 self._seenparams.add(pname)
21601 7ff01befc7ec bundle2: track life cycle of parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 21600 diff changeset	913 # status of the part's generation:
7ff01befc7ec bundle2: track life cycle of parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 21600 diff changeset	914 # - None: not started,
7ff01befc7ec bundle2: track life cycle of parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 21600 diff changeset	915 # - False: currently generated,
7ff01befc7ec bundle2: track life cycle of parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 21600 diff changeset	916 # - True: generation done.
7ff01befc7ec bundle2: track life cycle of parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 21600 diff changeset	917 self._generated = None
23602 4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API Eric Sumner <ericsumner@fb.com> parents: 23598 diff changeset	918 self.mandatory = mandatory
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	919
31651 1f51b4658f21 bundle2: implement a basic __repr__ for bundle2 part Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31650 diff changeset	920 def __repr__(self):
1f51b4658f21 bundle2: implement a basic __repr__ for bundle2 part Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31650 diff changeset	921 cls = "%s.%s" % (self.__class__.__module__, self.__class__.__name__)
1f51b4658f21 bundle2: implement a basic __repr__ for bundle2 part Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31650 diff changeset	922 return ('<%s object at %x; id: %s; type: %s; mandatory: %s>'
1f51b4658f21 bundle2: implement a basic __repr__ for bundle2 part Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31650 diff changeset	923 % (cls, id(self), self.id, self.type, self.mandatory))
1f51b4658f21 bundle2: implement a basic __repr__ for bundle2 part Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 31650 diff changeset	924
24972 2ec894582ea2 bundle2: add a 'copy' method on parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 24922 diff changeset	925 def copy(self):
2ec894582ea2 bundle2: add a 'copy' method on parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 24922 diff changeset	926 """return a copy of the part
2ec894582ea2 bundle2: add a 'copy' method on parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 24922 diff changeset	927
2ec894582ea2 bundle2: add a 'copy' method on parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 24922 diff changeset	928 The new part have the very same content but no partid assigned yet.
2ec894582ea2 bundle2: add a 'copy' method on parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 24922 diff changeset	929 Parts with generated data cannot be copied."""
2ec894582ea2 bundle2: add a 'copy' method on parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 24922 diff changeset	930 assert not util.safehasattr(self.data, 'next')
2ec894582ea2 bundle2: add a 'copy' method on parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 24922 diff changeset	931 return self.__class__(self.type, self._mandatoryparams,
2ec894582ea2 bundle2: add a 'copy' method on parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 24922 diff changeset	932 self._advisoryparams, self._data, self.mandatory)
2ec894582ea2 bundle2: add a 'copy' method on parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 24922 diff changeset	933
21604 c399bf961cb9 bundle2: the ability to set ``data`` attribute of the part is now official Pierre-Yves David <pierre-yves.david@fb.com> parents: 21603 diff changeset	934 # methods used to defines the part content
28534 52a4ad62b006 cleanup: use modern @property/@foo.setter property specification Augie Fackler <augie@google.com> parents: 28406 diff changeset	935 @property
52a4ad62b006 cleanup: use modern @property/@foo.setter property specification Augie Fackler <augie@google.com> parents: 28406 diff changeset	936 def data(self):
52a4ad62b006 cleanup: use modern @property/@foo.setter property specification Augie Fackler <augie@google.com> parents: 28406 diff changeset	937 return self._data
52a4ad62b006 cleanup: use modern @property/@foo.setter property specification Augie Fackler <augie@google.com> parents: 28406 diff changeset	938
52a4ad62b006 cleanup: use modern @property/@foo.setter property specification Augie Fackler <augie@google.com> parents: 28406 diff changeset	939 @data.setter
52a4ad62b006 cleanup: use modern @property/@foo.setter property specification Augie Fackler <augie@google.com> parents: 28406 diff changeset	940 def data(self, data):
21604 c399bf961cb9 bundle2: the ability to set ``data`` attribute of the part is now official Pierre-Yves David <pierre-yves.david@fb.com> parents: 21603 diff changeset	941 if self._generated is not None:
21618 7568f5c1c801 bundle2: move exception classes into the error module Pierre-Yves David <pierre-yves.david@fb.com> parents: 21617 diff changeset	942 raise error.ReadOnlyPartError('part is being generated')
21604 c399bf961cb9 bundle2: the ability to set ``data`` attribute of the part is now official Pierre-Yves David <pierre-yves.david@fb.com> parents: 21603 diff changeset	943 self._data = data
c399bf961cb9 bundle2: the ability to set ``data`` attribute of the part is now official Pierre-Yves David <pierre-yves.david@fb.com> parents: 21603 diff changeset	944
21605 f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	945 @property
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	946 def mandatoryparams(self):
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	947 # make it an immutable tuple to force people through ``addparam``
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	948 return tuple(self._mandatoryparams)
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	949
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	950 @property
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	951 def advisoryparams(self):
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	952 # make it an immutable tuple to force people through ``addparam``
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	953 return tuple(self._advisoryparams)
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	954
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	955 def addparam(self, name, value='', mandatory=True):
32640 6d055cd6669f bundle2: add documention to 'part.addparams' Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32426 diff changeset	956 """add a parameter to the part
6d055cd6669f bundle2: add documention to 'part.addparams' Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32426 diff changeset	957
6d055cd6669f bundle2: add documention to 'part.addparams' Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32426 diff changeset	958 If 'mandatory' is set to True, the remote handler must claim support
6d055cd6669f bundle2: add documention to 'part.addparams' Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32426 diff changeset	959 for this parameter or the unbundling will be aborted.
6d055cd6669f bundle2: add documention to 'part.addparams' Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32426 diff changeset	960
6d055cd6669f bundle2: add documention to 'part.addparams' Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32426 diff changeset	961 The 'name' and 'value' cannot exceed 255 bytes each.
6d055cd6669f bundle2: add documention to 'part.addparams' Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32426 diff changeset	962 """
21605 f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	963 if self._generated is not None:
21618 7568f5c1c801 bundle2: move exception classes into the error module Pierre-Yves David <pierre-yves.david@fb.com> parents: 21617 diff changeset	964 raise error.ReadOnlyPartError('part is being generated')
21607 054fa5176fa7 bundle2: forbid duplicate parameter keys Pierre-Yves David <pierre-yves.david@fb.com> parents: 21606 diff changeset	965 if name in self._seenparams:
054fa5176fa7 bundle2: forbid duplicate parameter keys Pierre-Yves David <pierre-yves.david@fb.com> parents: 21606 diff changeset	966 raise ValueError('duplicated params: %s' % name)
054fa5176fa7 bundle2: forbid duplicate parameter keys Pierre-Yves David <pierre-yves.david@fb.com> parents: 21606 diff changeset	967 self._seenparams.add(name)
21605 f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	968 params = self._advisoryparams
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	969 if mandatory:
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	970 params = self._mandatoryparams
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	971 params.append((name, value))
f9dabfceb259 bundle2: introduce a ``addparam`` method on part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21604 diff changeset	972
21601 7ff01befc7ec bundle2: track life cycle of parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 21600 diff changeset	973 # methods used to generates the bundle2 stream
25805 b44ee346211f bundle2: add debug output for part generation Pierre-Yves David <pierre-yves.david@fb.com> parents: 25804 diff changeset	974 def getchunks(self, ui):
21601 7ff01befc7ec bundle2: track life cycle of parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 21600 diff changeset	975 if self._generated is not None:
32426 4dbef666c6c9 bundle2: use ProgrammingError Jun Wu <quark@fb.com> parents: 32255 diff changeset	976 raise error.ProgrammingError('part can only be consumed once')
21601 7ff01befc7ec bundle2: track life cycle of parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 21600 diff changeset	977 self._generated = False
25807 21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	978
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	979 if ui.debugflag:
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	980 msg = ['bundle2-output-part: "%s"' % self.type]
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	981 if not self.mandatory:
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	982 msg.append(' (advisory)')
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	983 nbmp = len(self.mandatoryparams)
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	984 nbap = len(self.advisoryparams)
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	985 if nbmp or nbap:
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	986 msg.append(' (params:')
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	987 if nbmp:
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	988 msg.append(' %i mandatory' % nbmp)
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	989 if nbap:
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	990 msg.append(' %i advisory' % nbmp)
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	991 msg.append(')')
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	992 if not self.data:
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	993 msg.append(' empty payload')
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	994 elif util.safehasattr(self.data, 'next'):
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	995 msg.append(' streamed payload')
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	996 else:
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	997 msg.append(' %i bytes payload' % len(self.data))
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	998 msg.append('\n')
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	999 ui.debug(''.join(msg))
21a25fb81d2c bundle2: add generic debug output regarding generated parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 25806 diff changeset	1000
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1001 #### header
23602 4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API Eric Sumner <ericsumner@fb.com> parents: 23598 diff changeset	1002 if self.mandatory:
4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API Eric Sumner <ericsumner@fb.com> parents: 23598 diff changeset	1003 parttype = self.type.upper()
4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API Eric Sumner <ericsumner@fb.com> parents: 23598 diff changeset	1004 else:
4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API Eric Sumner <ericsumner@fb.com> parents: 23598 diff changeset	1005 parttype = self.type.lower()
34494 373ca5103ba8 bundle2: convert ints to strings using pycompat.bytestring() Augie Fackler <augie@google.com> parents: 34476 diff changeset	1006 outdebug(ui, 'part %s: "%s"' % (pycompat.bytestr(self.id), parttype))
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1007 ## parttype
23602 4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API Eric Sumner <ericsumner@fb.com> parents: 23598 diff changeset	1008 header = [_pack(_fparttypesize, len(parttype)),
4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API Eric Sumner <ericsumner@fb.com> parents: 23598 diff changeset	1009 parttype, _pack(_fpartid, self.id),
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	1010 ]
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1011 ## parameters
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1012 # count
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1013 manpar = self.mandatoryparams
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1014 advpar = self.advisoryparams
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1015 header.append(_pack(_fpartparamcount, len(manpar), len(advpar)))
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1016 # size
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1017 parsizes = []
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1018 for key, value in manpar:
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1019 parsizes.append(len(key))
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1020 parsizes.append(len(value))
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1021 for key, value in advpar:
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1022 parsizes.append(len(key))
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1023 parsizes.append(len(value))
34455 da7c285ec6da bundle2: use modern Python division Augie Fackler <augie@google.com> parents: 34448 diff changeset	1024 paramsizes = _pack(_makefpartparamsizes(len(parsizes) // 2), *parsizes)
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1025 header.append(paramsizes)
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1026 # key, value
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1027 for key, value in manpar:
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1028 header.append(key)
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1029 header.append(value)
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1030 for key, value in advpar:
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1031 header.append(key)
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1032 header.append(value)
9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1033 ## finalize header
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	1034 headerchunk = ''.join(header)
25805 b44ee346211f bundle2: add debug output for part generation Pierre-Yves David <pierre-yves.david@fb.com> parents: 25804 diff changeset	1035 outdebug(ui, 'header chunk size: %i' % len(headerchunk))
20856 8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	1036 yield _pack(_fpartheadersize, len(headerchunk))
8a6a86c9a5b5 bundle2: support bundling of empty part (with a type) Pierre-Yves David <pierre-yves.david@fb.com> parents: 20844 diff changeset	1037 yield headerchunk
20877 9e9e3a4e9261 bundle2: part params Pierre-Yves David <pierre-yves.david@fb.com> parents: 20876 diff changeset	1038 ## payload
23067 420a051616ce bundle2: transmit exception during part generation Pierre-Yves David <pierre-yves.david@fb.com> parents: 23066 diff changeset	1039 try:
420a051616ce bundle2: transmit exception during part generation Pierre-Yves David <pierre-yves.david@fb.com> parents: 23066 diff changeset	1040 for chunk in self._payloadchunks():
25805 b44ee346211f bundle2: add debug output for part generation Pierre-Yves David <pierre-yves.david@fb.com> parents: 25804 diff changeset	1041 outdebug(ui, 'payload chunk size: %i' % len(chunk))
23067 420a051616ce bundle2: transmit exception during part generation Pierre-Yves David <pierre-yves.david@fb.com> parents: 23066 diff changeset	1042 yield _pack(_fpayloadsize, len(chunk))
420a051616ce bundle2: transmit exception during part generation Pierre-Yves David <pierre-yves.david@fb.com> parents: 23066 diff changeset	1043 yield chunk
26740 4bc3707f3e67 bundle2: don't try to recover from a GeneratorExit (issue4785) Augie Fackler <augie@google.com> parents: 26503 diff changeset	1044 except GeneratorExit:
4bc3707f3e67 bundle2: don't try to recover from a GeneratorExit (issue4785) Augie Fackler <augie@google.com> parents: 26503 diff changeset	1045 # GeneratorExit means that nobody is listening for our
4bc3707f3e67 bundle2: don't try to recover from a GeneratorExit (issue4785) Augie Fackler <augie@google.com> parents: 26503 diff changeset	1046 # results anyway, so just bail quickly rather than trying
4bc3707f3e67 bundle2: don't try to recover from a GeneratorExit (issue4785) Augie Fackler <augie@google.com> parents: 26503 diff changeset	1047 # to produce an error part.
4bc3707f3e67 bundle2: don't try to recover from a GeneratorExit (issue4785) Augie Fackler <augie@google.com> parents: 26503 diff changeset	1048 ui.debug('bundle2-generatorexit\n')
4bc3707f3e67 bundle2: don't try to recover from a GeneratorExit (issue4785) Augie Fackler <augie@google.com> parents: 26503 diff changeset	1049 raise
26214 328739ea70c3 global: mass rewrite to use modern exception syntax Gregory Szorc <gregory.szorc@gmail.com> parents: 26195 diff changeset	1050 except BaseException as exc:
34465 1d5e497c08b3 py3: convert arbitrary exception object to byte string more reliably Yuya Nishihara <yuya@tcha.org> parents: 34459 diff changeset	1051 bexc = util.forcebytestr(exc)
23067 420a051616ce bundle2: transmit exception during part generation Pierre-Yves David <pierre-yves.david@fb.com> parents: 23066 diff changeset	1052 # backup exception data for later
25808 deed9c6b12d2 bundle2: add generic debug output regarding generated interruption Pierre-Yves David <pierre-yves.david@fb.com> parents: 25807 diff changeset	1053 ui.debug('bundle2-input-stream-interrupt: encoding exception %s'
34459 9f862fe7a8bd bundle2: obtain repr() of exception in a python3-safe way Augie Fackler <augie@google.com> parents: 34458 diff changeset	1054 % bexc)
32969 76f9a0009b4b pycompat: extract helper to raise exception with traceback Yuya Nishihara <yuya@tcha.org> parents: 32803 diff changeset	1055 tb = sys.exc_info()[2]
34459 9f862fe7a8bd bundle2: obtain repr() of exception in a python3-safe way Augie Fackler <augie@google.com> parents: 34458 diff changeset	1056 msg = 'unexpected error: %s' % bexc
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1057 interpart = bundlepart('error:abort', [('message', msg)],
23602 4440c7cc3728 bundle2.bundlepart: make mandatory part flag explicit in API Eric Sumner <ericsumner@fb.com> parents: 23598 diff changeset	1058 mandatory=False)
23067 420a051616ce bundle2: transmit exception during part generation Pierre-Yves David <pierre-yves.david@fb.com> parents: 23066 diff changeset	1059 interpart.id = 0
420a051616ce bundle2: transmit exception during part generation Pierre-Yves David <pierre-yves.david@fb.com> parents: 23066 diff changeset	1060 yield _pack(_fpayloadsize, -1)
25805 b44ee346211f bundle2: add debug output for part generation Pierre-Yves David <pierre-yves.david@fb.com> parents: 25804 diff changeset	1061 for chunk in interpart.getchunks(ui=ui):
23067 420a051616ce bundle2: transmit exception during part generation Pierre-Yves David <pierre-yves.david@fb.com> parents: 23066 diff changeset	1062 yield chunk
25805 b44ee346211f bundle2: add debug output for part generation Pierre-Yves David <pierre-yves.david@fb.com> parents: 25804 diff changeset	1063 outdebug(ui, 'closing payload chunk')
23067 420a051616ce bundle2: transmit exception during part generation Pierre-Yves David <pierre-yves.david@fb.com> parents: 23066 diff changeset	1064 # abort current part payload
420a051616ce bundle2: transmit exception during part generation Pierre-Yves David <pierre-yves.david@fb.com> parents: 23066 diff changeset	1065 yield _pack(_fpayloadsize, 0)
32969 76f9a0009b4b pycompat: extract helper to raise exception with traceback Yuya Nishihara <yuya@tcha.org> parents: 32803 diff changeset	1066 pycompat.raisewithtb(exc, tb)
21000 4cae06ae1562 bundle2: extract a _payloadchunks method for part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20998 diff changeset	1067 # end of payload
25805 b44ee346211f bundle2: add debug output for part generation Pierre-Yves David <pierre-yves.david@fb.com> parents: 25804 diff changeset	1068 outdebug(ui, 'closing payload chunk')
21000 4cae06ae1562 bundle2: extract a _payloadchunks method for part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20998 diff changeset	1069 yield _pack(_fpayloadsize, 0)
21601 7ff01befc7ec bundle2: track life cycle of parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 21600 diff changeset	1070 self._generated = True
21000 4cae06ae1562 bundle2: extract a _payloadchunks method for part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20998 diff changeset	1071
4cae06ae1562 bundle2: extract a _payloadchunks method for part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20998 diff changeset	1072 def _payloadchunks(self):
4cae06ae1562 bundle2: extract a _payloadchunks method for part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20998 diff changeset	1073 """yield chunks of a the part payload
4cae06ae1562 bundle2: extract a _payloadchunks method for part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20998 diff changeset	1074
4cae06ae1562 bundle2: extract a _payloadchunks method for part Pierre-Yves David <pierre-yves.david@fb.com> parents: 20998 diff changeset	1075 Exists to handle the different methods to provide data to a part."""
20876 ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	1076 # we only support fixed size data now.
ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	1077 # This will be improved in the future.
34456 5ae35a1347fd bundle2: look for __next__ as well as next to identify iterators Augie Fackler <augie@google.com> parents: 34455 diff changeset	1078 if (util.safehasattr(self.data, 'next')
5ae35a1347fd bundle2: look for __next__ as well as next to identify iterators Augie Fackler <augie@google.com> parents: 34455 diff changeset	1079 or util.safehasattr(self.data, '__next__')):
21001 c93bb6a08fa1 bundle2: support chunk iterator as part data Pierre-Yves David <pierre-yves.david@fb.com> parents: 21000 diff changeset	1080 buff = util.chunkbuffer(self.data)
c93bb6a08fa1 bundle2: support chunk iterator as part data Pierre-Yves David <pierre-yves.david@fb.com> parents: 21000 diff changeset	1081 chunk = buff.read(preferedchunksize)
c93bb6a08fa1 bundle2: support chunk iterator as part data Pierre-Yves David <pierre-yves.david@fb.com> parents: 21000 diff changeset	1082 while chunk:
c93bb6a08fa1 bundle2: support chunk iterator as part data Pierre-Yves David <pierre-yves.david@fb.com> parents: 21000 diff changeset	1083 yield chunk
c93bb6a08fa1 bundle2: support chunk iterator as part data Pierre-Yves David <pierre-yves.david@fb.com> parents: 21000 diff changeset	1084 chunk = buff.read(preferedchunksize)
c93bb6a08fa1 bundle2: support chunk iterator as part data Pierre-Yves David <pierre-yves.david@fb.com> parents: 21000 diff changeset	1085 elif len(self.data):
20876 ddd56f3eb786 bundle2: support for bundling and unbundling payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 20864 diff changeset	1086 yield self.data
20802 520df53ad26a bundle2: a very first version of bundle2 unbundler Pierre-Yves David <pierre-yves.david@fb.com> parents: 20801 diff changeset	1087
23066 ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1088
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1089 flaginterrupt = -1
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1090
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1091 class interrupthandler(unpackermixin):
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1092 """read one part and process it with restricted capability
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1093
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1094 This allows to transmit exception raised on the producer size during part
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1095 iteration while the consumer is reading a part.
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1096
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1097 Part processed in this manner only have access to a ui object,"""
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1098
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1099 def __init__(self, ui, fp):
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1100 super(interrupthandler, self).__init__(fp)
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1101 self.ui = ui
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1102
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1103 def _readpartheader(self):
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1104 """reads a part header size and return the bytes blob
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1105
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1106 returns None if empty"""
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1107 headersize = self._unpack(_fpartheadersize)[0]
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1108 if headersize < 0:
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1109 raise error.BundleValueError('negative part header size: %i'
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1110 % headersize)
25802 64dd8f25fcf4 bundle2: introduce a specific function for debug messages while unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 25801 diff changeset	1111 indebug(self.ui, 'part header size: %i\n' % headersize)
23066 ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1112 if headersize:
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1113 return self._readexact(headersize)
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1114 return None
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1115
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1116 def __call__(self):
25819 8f7137a85a0e bundle2: add generic debug output regarding processed interruption Pierre-Yves David <pierre-yves.david@fb.com> parents: 25818 diff changeset	1117
8f7137a85a0e bundle2: add generic debug output regarding processed interruption Pierre-Yves David <pierre-yves.david@fb.com> parents: 25818 diff changeset	1118 self.ui.debug('bundle2-input-stream-interrupt:'
8f7137a85a0e bundle2: add generic debug output regarding processed interruption Pierre-Yves David <pierre-yves.david@fb.com> parents: 25818 diff changeset	1119 ' opening out of band context\n')
25804 697d8953b04d bundle2: handle new line in 'indebug' function Pierre-Yves David <pierre-yves.david@fb.com> parents: 25803 diff changeset	1120 indebug(self.ui, 'bundle2 stream interruption, looking for a part.')
23066 ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1121 headerblock = self._readpartheader()
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1122 if headerblock is None:
25804 697d8953b04d bundle2: handle new line in 'indebug' function Pierre-Yves David <pierre-yves.david@fb.com> parents: 25803 diff changeset	1123 indebug(self.ui, 'no part found during interruption.')
23066 ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1124 return
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1125 part = unbundlepart(self.ui, headerblock, self._fp)
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1126 op = interruptoperation(self.ui)
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1127 _processpart(op, part)
25819 8f7137a85a0e bundle2: add generic debug output regarding processed interruption Pierre-Yves David <pierre-yves.david@fb.com> parents: 25818 diff changeset	1128 self.ui.debug('bundle2-input-stream-interrupt:'
8f7137a85a0e bundle2: add generic debug output regarding processed interruption Pierre-Yves David <pierre-yves.david@fb.com> parents: 25818 diff changeset	1129 ' closing out of band context\n')
23066 ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1130
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1131 class interruptoperation(object):
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1132 """A limited operation to be use by part handler during interruption
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1133
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1134 It only have access to an ui object.
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1135 """
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1136
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1137 def __init__(self, ui):
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1138 self.ui = ui
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1139 self.reply = None
25135 e530cde6d115 bundle2: disable ouput capture unless we use http (issue4613 issue4615) Pierre-Yves David <pierre-yves.david@fb.com> parents: 25056 diff changeset	1140 self.captureoutput = False
23066 ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1141
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1142 @property
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1143 def repo(self):
32426 4dbef666c6c9 bundle2: use ProgrammingError Jun Wu <quark@fb.com> parents: 32255 diff changeset	1144 raise error.ProgrammingError('no repo access from stream interruption')
23066 ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1145
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1146 def gettransaction(self):
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1147 raise TransactionUnavailable('no repo access from stream interruption')
ad144882318d bundle2: add a interrupt mechanism Pierre-Yves David <pierre-yves.david@fb.com> parents: 23029 diff changeset	1148
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1149 class unbundlepart(unpackermixin):
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1150 """a bundle part read from a bundle"""
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1151
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1152 def __init__(self, ui, header, fp):
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1153 super(unbundlepart, self).__init__(fp)
32668 a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1154 self._seekable = (util.safehasattr(fp, 'seek') and
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1155 util.safehasattr(fp, 'tell'))
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1156 self.ui = ui
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1157 # unbundle state attr
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1158 self._headerdata = header
21015 14dd49260246 bundle2: move the fromheader closure into the class itself Pierre-Yves David <pierre-yves.david@fb.com> parents: 21014 diff changeset	1159 self._headeroffset = 0
21019 3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1160 self._initialized = False
3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1161 self.consumed = False
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1162 # part data
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1163 self.id = None
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1164 self.type = None
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1165 self.mandatoryparams = None
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1166 self.advisoryparams = None
21610 d6056805f8f4 bundle2: introduce a ``params`` dictionary on unbundled parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 21609 diff changeset	1167 self.params = None
21612 f221eb0531d9 bundle2: expose mandatory params in a mandatorykeys attribute Pierre-Yves David <pierre-yves.david@fb.com> parents: 21611 diff changeset	1168 self.mandatorykeys = ()
21019 3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1169 self._payloadstream = None
3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1170 self._readheader()
23597 94b25d71dd0f bundle2.unbundlepart: decouple mandatory from parttype Eric Sumner <ericsumner@fb.com> parents: 23439 diff changeset	1171 self._mandatory = None
24047 7eb26415bef6 bundle2.unbundlepart: keep an index of chunks and their locations Eric Sumner <ericsumner@fb.com> parents: 24046 diff changeset	1172 self._chunkindex = [] #(payload, file) position tuples for chunk starts
24048 c7601086338a bundle2.unbundlepart: tell() implementation Eric Sumner <ericsumner@fb.com> parents: 24047 diff changeset	1173 self._pos = 0
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1174
21015 14dd49260246 bundle2: move the fromheader closure into the class itself Pierre-Yves David <pierre-yves.david@fb.com> parents: 21014 diff changeset	1175 def _fromheader(self, size):
14dd49260246 bundle2: move the fromheader closure into the class itself Pierre-Yves David <pierre-yves.david@fb.com> parents: 21014 diff changeset	1176 """return the next <size> byte from the header"""
14dd49260246 bundle2: move the fromheader closure into the class itself Pierre-Yves David <pierre-yves.david@fb.com> parents: 21014 diff changeset	1177 offset = self._headeroffset
14dd49260246 bundle2: move the fromheader closure into the class itself Pierre-Yves David <pierre-yves.david@fb.com> parents: 21014 diff changeset	1178 data = self._headerdata[offset:(offset + size)]
21019 3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1179 self._headeroffset = offset + size
21015 14dd49260246 bundle2: move the fromheader closure into the class itself Pierre-Yves David <pierre-yves.david@fb.com> parents: 21014 diff changeset	1180 return data
14dd49260246 bundle2: move the fromheader closure into the class itself Pierre-Yves David <pierre-yves.david@fb.com> parents: 21014 diff changeset	1181
21016 b477afb1c81e bundle2: move unpackheader closure into the class Pierre-Yves David <pierre-yves.david@fb.com> parents: 21015 diff changeset	1182 def _unpackheader(self, format):
b477afb1c81e bundle2: move unpackheader closure into the class Pierre-Yves David <pierre-yves.david@fb.com> parents: 21015 diff changeset	1183 """read given format from header
b477afb1c81e bundle2: move unpackheader closure into the class Pierre-Yves David <pierre-yves.david@fb.com> parents: 21015 diff changeset	1184
b477afb1c81e bundle2: move unpackheader closure into the class Pierre-Yves David <pierre-yves.david@fb.com> parents: 21015 diff changeset	1185 This automatically compute the size of the format to read."""
b477afb1c81e bundle2: move unpackheader closure into the class Pierre-Yves David <pierre-yves.david@fb.com> parents: 21015 diff changeset	1186 data = self._fromheader(struct.calcsize(format))
b477afb1c81e bundle2: move unpackheader closure into the class Pierre-Yves David <pierre-yves.david@fb.com> parents: 21015 diff changeset	1187 return _unpack(format, data)
b477afb1c81e bundle2: move unpackheader closure into the class Pierre-Yves David <pierre-yves.david@fb.com> parents: 21015 diff changeset	1188
21608 3cb96ca90c17 bundle2: introduce an ``_initparams`` method Pierre-Yves David <pierre-yves.david@fb.com> parents: 21607 diff changeset	1189 def _initparams(self, mandatoryparams, advisoryparams):
3cb96ca90c17 bundle2: introduce an ``_initparams`` method Pierre-Yves David <pierre-yves.david@fb.com> parents: 21607 diff changeset	1190 """internal function to setup all logic related parameters"""
21609 63cc2594ef8a bundle2: make sure unbundled part param are read-only Pierre-Yves David <pierre-yves.david@fb.com> parents: 21608 diff changeset	1191 # make it read only to prevent people touching it by mistake.
63cc2594ef8a bundle2: make sure unbundled part param are read-only Pierre-Yves David <pierre-yves.david@fb.com> parents: 21608 diff changeset	1192 self.mandatoryparams = tuple(mandatoryparams)
63cc2594ef8a bundle2: make sure unbundled part param are read-only Pierre-Yves David <pierre-yves.david@fb.com> parents: 21608 diff changeset	1193 self.advisoryparams = tuple(advisoryparams)
21610 d6056805f8f4 bundle2: introduce a ``params`` dictionary on unbundled parts Pierre-Yves David <pierre-yves.david@fb.com> parents: 21609 diff changeset	1194 # user friendly UI
30352 6215b5537ba5 bundle2: use a sorted dict for holding parameters Gregory Szorc <gregory.szorc@gmail.com> parents: 30044 diff changeset	1195 self.params = util.sortdict(self.mandatoryparams)
6215b5537ba5 bundle2: use a sorted dict for holding parameters Gregory Szorc <gregory.szorc@gmail.com> parents: 30044 diff changeset	1196 self.params.update(self.advisoryparams)
21612 f221eb0531d9 bundle2: expose mandatory params in a mandatorykeys attribute Pierre-Yves David <pierre-yves.david@fb.com> parents: 21611 diff changeset	1197 self.mandatorykeys = frozenset(p[0] for p in mandatoryparams)
21608 3cb96ca90c17 bundle2: introduce an ``_initparams`` method Pierre-Yves David <pierre-yves.david@fb.com> parents: 21607 diff changeset	1198
24047 7eb26415bef6 bundle2.unbundlepart: keep an index of chunks and their locations Eric Sumner <ericsumner@fb.com> parents: 24046 diff changeset	1199 def _payloadchunks(self, chunknum=0):
7eb26415bef6 bundle2.unbundlepart: keep an index of chunks and their locations Eric Sumner <ericsumner@fb.com> parents: 24046 diff changeset	1200 '''seek to specified chunk and start yielding data'''
7eb26415bef6 bundle2.unbundlepart: keep an index of chunks and their locations Eric Sumner <ericsumner@fb.com> parents: 24046 diff changeset	1201 if len(self._chunkindex) == 0:
7eb26415bef6 bundle2.unbundlepart: keep an index of chunks and their locations Eric Sumner <ericsumner@fb.com> parents: 24046 diff changeset	1202 assert chunknum == 0, 'Must start with chunk 0'
32668 a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1203 self._chunkindex.append((0, self._tellfp()))
24047 7eb26415bef6 bundle2.unbundlepart: keep an index of chunks and their locations Eric Sumner <ericsumner@fb.com> parents: 24046 diff changeset	1204 else:
7eb26415bef6 bundle2.unbundlepart: keep an index of chunks and their locations Eric Sumner <ericsumner@fb.com> parents: 24046 diff changeset	1205 assert chunknum < len(self._chunkindex), \
7eb26415bef6 bundle2.unbundlepart: keep an index of chunks and their locations Eric Sumner <ericsumner@fb.com> parents: 24046 diff changeset	1206 'Unknown chunk %d' % chunknum
32668 a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1207 self._seekfp(self._chunkindex[chunknum][1])
24047 7eb26415bef6 bundle2.unbundlepart: keep an index of chunks and their locations Eric Sumner <ericsumner@fb.com> parents: 24046 diff changeset	1208
7eb26415bef6 bundle2.unbundlepart: keep an index of chunks and their locations Eric Sumner <ericsumner@fb.com> parents: 24046 diff changeset	1209 pos = self._chunkindex[chunknum][0]
24046 9881a1437799 bundle2.unbundlepart: raise payloadchunks from a closure to a method Eric Sumner <ericsumner@fb.com> parents: 24038 diff changeset	1210 payloadsize = self._unpack(_fpayloadsize)[0]
25804 697d8953b04d bundle2: handle new line in 'indebug' function Pierre-Yves David <pierre-yves.david@fb.com> parents: 25803 diff changeset	1211 indebug(self.ui, 'payload chunk size: %i' % payloadsize)
24046 9881a1437799 bundle2.unbundlepart: raise payloadchunks from a closure to a method Eric Sumner <ericsumner@fb.com> parents: 24038 diff changeset	1212 while payloadsize:
9881a1437799 bundle2.unbundlepart: raise payloadchunks from a closure to a method Eric Sumner <ericsumner@fb.com> parents: 24038 diff changeset	1213 if payloadsize == flaginterrupt:
9881a1437799 bundle2.unbundlepart: raise payloadchunks from a closure to a method Eric Sumner <ericsumner@fb.com> parents: 24038 diff changeset	1214 # interruption detection, the handler will now read a
9881a1437799 bundle2.unbundlepart: raise payloadchunks from a closure to a method Eric Sumner <ericsumner@fb.com> parents: 24038 diff changeset	1215 # single part and process it.
9881a1437799 bundle2.unbundlepart: raise payloadchunks from a closure to a method Eric Sumner <ericsumner@fb.com> parents: 24038 diff changeset	1216 interrupthandler(self.ui, self._fp)()
9881a1437799 bundle2.unbundlepart: raise payloadchunks from a closure to a method Eric Sumner <ericsumner@fb.com> parents: 24038 diff changeset	1217 elif payloadsize < 0:
9881a1437799 bundle2.unbundlepart: raise payloadchunks from a closure to a method Eric Sumner <ericsumner@fb.com> parents: 24038 diff changeset	1218 msg = 'negative payload chunk size: %i' % payloadsize
9881a1437799 bundle2.unbundlepart: raise payloadchunks from a closure to a method Eric Sumner <ericsumner@fb.com> parents: 24038 diff changeset	1219 raise error.BundleValueError(msg)
9881a1437799 bundle2.unbundlepart: raise payloadchunks from a closure to a method Eric Sumner <ericsumner@fb.com> parents: 24038 diff changeset	1220 else:
24047 7eb26415bef6 bundle2.unbundlepart: keep an index of chunks and their locations Eric Sumner <ericsumner@fb.com> parents: 24046 diff changeset	1221 result = self._readexact(payloadsize)
7eb26415bef6 bundle2.unbundlepart: keep an index of chunks and their locations Eric Sumner <ericsumner@fb.com> parents: 24046 diff changeset	1222 chunknum += 1
7eb26415bef6 bundle2.unbundlepart: keep an index of chunks and their locations Eric Sumner <ericsumner@fb.com> parents: 24046 diff changeset	1223 pos += payloadsize
7eb26415bef6 bundle2.unbundlepart: keep an index of chunks and their locations Eric Sumner <ericsumner@fb.com> parents: 24046 diff changeset	1224 if chunknum == len(self._chunkindex):
32668 a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1225 self._chunkindex.append((pos, self._tellfp()))
24047 7eb26415bef6 bundle2.unbundlepart: keep an index of chunks and their locations Eric Sumner <ericsumner@fb.com> parents: 24046 diff changeset	1226 yield result
24046 9881a1437799 bundle2.unbundlepart: raise payloadchunks from a closure to a method Eric Sumner <ericsumner@fb.com> parents: 24038 diff changeset	1227 payloadsize = self._unpack(_fpayloadsize)[0]
25804 697d8953b04d bundle2: handle new line in 'indebug' function Pierre-Yves David <pierre-yves.david@fb.com> parents: 25803 diff changeset	1228 indebug(self.ui, 'payload chunk size: %i' % payloadsize)
24046 9881a1437799 bundle2.unbundlepart: raise payloadchunks from a closure to a method Eric Sumner <ericsumner@fb.com> parents: 24038 diff changeset	1229
24049 f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1230 def _findchunk(self, pos):
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1231 '''for a given payload position, return a chunk number and offset'''
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1232 for chunk, (ppos, fpos) in enumerate(self._chunkindex):
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1233 if ppos == pos:
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1234 return chunk, 0
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1235 elif ppos > pos:
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1236 return chunk - 1, pos - self._chunkindex[chunk - 1][0]
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1237 raise ValueError('Unknown chunk')
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1238
21019 3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1239 def _readheader(self):
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1240 """read the header and setup the object"""
21016 b477afb1c81e bundle2: move unpackheader closure into the class Pierre-Yves David <pierre-yves.david@fb.com> parents: 21015 diff changeset	1241 typesize = self._unpackheader(_fparttypesize)[0]
21015 14dd49260246 bundle2: move the fromheader closure into the class itself Pierre-Yves David <pierre-yves.david@fb.com> parents: 21014 diff changeset	1242 self.type = self._fromheader(typesize)
25804 697d8953b04d bundle2: handle new line in 'indebug' function Pierre-Yves David <pierre-yves.david@fb.com> parents: 25803 diff changeset	1243 indebug(self.ui, 'part type: "%s"' % self.type)
21016 b477afb1c81e bundle2: move unpackheader closure into the class Pierre-Yves David <pierre-yves.david@fb.com> parents: 21015 diff changeset	1244 self.id = self._unpackheader(_fpartid)[0]
34494 373ca5103ba8 bundle2: convert ints to strings using pycompat.bytestring() Augie Fackler <augie@google.com> parents: 34476 diff changeset	1245 indebug(self.ui, 'part id: "%s"' % pycompat.bytestr(self.id))
23597 94b25d71dd0f bundle2.unbundlepart: decouple mandatory from parttype Eric Sumner <ericsumner@fb.com> parents: 23439 diff changeset	1246 # extract mandatory bit from type
94b25d71dd0f bundle2.unbundlepart: decouple mandatory from parttype Eric Sumner <ericsumner@fb.com> parents: 23439 diff changeset	1247 self.mandatory = (self.type != self.type.lower())
94b25d71dd0f bundle2.unbundlepart: decouple mandatory from parttype Eric Sumner <ericsumner@fb.com> parents: 23439 diff changeset	1248 self.type = self.type.lower()
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1249 ## reading parameters
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1250 # param count
21016 b477afb1c81e bundle2: move unpackheader closure into the class Pierre-Yves David <pierre-yves.david@fb.com> parents: 21015 diff changeset	1251 mancount, advcount = self._unpackheader(_fpartparamcount)
25804 697d8953b04d bundle2: handle new line in 'indebug' function Pierre-Yves David <pierre-yves.david@fb.com> parents: 25803 diff changeset	1252 indebug(self.ui, 'part parameters: %i' % (mancount + advcount))
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1253 # param size
21016 b477afb1c81e bundle2: move unpackheader closure into the class Pierre-Yves David <pierre-yves.david@fb.com> parents: 21015 diff changeset	1254 fparamsizes = _makefpartparamsizes(mancount + advcount)
b477afb1c81e bundle2: move unpackheader closure into the class Pierre-Yves David <pierre-yves.david@fb.com> parents: 21015 diff changeset	1255 paramsizes = self._unpackheader(fparamsizes)
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1256 # make it a list of couple again
34457 9a323773216c bundle2: work around zip() being lazy in Python 3 Augie Fackler <augie@google.com> parents: 34456 diff changeset	1257 paramsizes = list(zip(paramsizes[::2], paramsizes[1::2]))
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1258 # split mandatory from advisory
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1259 mansizes = paramsizes[:mancount]
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1260 advsizes = paramsizes[mancount:]
23139 e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues Mads Kiilerich <madski@unity3d.com> parents: 23067 diff changeset	1261 # retrieve param value
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1262 manparams = []
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1263 for key, value in mansizes:
21015 14dd49260246 bundle2: move the fromheader closure into the class itself Pierre-Yves David <pierre-yves.david@fb.com> parents: 21014 diff changeset	1264 manparams.append((self._fromheader(key), self._fromheader(value)))
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1265 advparams = []
a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1266 for key, value in advsizes:
21015 14dd49260246 bundle2: move the fromheader closure into the class itself Pierre-Yves David <pierre-yves.david@fb.com> parents: 21014 diff changeset	1267 advparams.append((self._fromheader(key), self._fromheader(value)))
21608 3cb96ca90c17 bundle2: introduce an ``_initparams`` method Pierre-Yves David <pierre-yves.david@fb.com> parents: 21607 diff changeset	1268 self._initparams(manparams, advparams)
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1269 ## part payload
24046 9881a1437799 bundle2.unbundlepart: raise payloadchunks from a closure to a method Eric Sumner <ericsumner@fb.com> parents: 24038 diff changeset	1270 self._payloadstream = util.chunkbuffer(self._payloadchunks())
21019 3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1271 # we read the data, tell it
3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1272 self._initialized = True
3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1273
3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1274 def read(self, size=None):
3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1275 """read payload data"""
3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1276 if not self._initialized:
3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1277 self._readheader()
3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1278 if size is None:
3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1279 data = self._payloadstream.read()
3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1280 else:
3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1281 data = self._payloadstream.read(size)
25818 76cba1ecf55f bundle2: add generic debug output regarding processed part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 25817 diff changeset	1282 self._pos += len(data)
21019 3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1283 if size is None or len(data) < size:
25818 76cba1ecf55f bundle2: add generic debug output regarding processed part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 25817 diff changeset	1284 if not self.consumed and self._pos:
76cba1ecf55f bundle2: add generic debug output regarding processed part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 25817 diff changeset	1285 self.ui.debug('bundle2-input-part: total payload size %i\n'
76cba1ecf55f bundle2: add generic debug output regarding processed part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 25817 diff changeset	1286 % self._pos)
21019 3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1287 self.consumed = True
3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1288 return data
3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1289
24048 c7601086338a bundle2.unbundlepart: tell() implementation Eric Sumner <ericsumner@fb.com> parents: 24047 diff changeset	1290 def tell(self):
c7601086338a bundle2.unbundlepart: tell() implementation Eric Sumner <ericsumner@fb.com> parents: 24047 diff changeset	1291 return self._pos
c7601086338a bundle2.unbundlepart: tell() implementation Eric Sumner <ericsumner@fb.com> parents: 24047 diff changeset	1292
24049 f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1293 def seek(self, offset, whence=0):
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1294 if whence == 0:
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1295 newpos = offset
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1296 elif whence == 1:
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1297 newpos = self._pos + offset
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1298 elif whence == 2:
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1299 if not self.consumed:
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1300 self.read()
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1301 newpos = self._chunkindex[-1][0] - offset
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1302 else:
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1303 raise ValueError('Unknown whence value: %r' % (whence,))
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1304
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1305 if newpos > self._chunkindex[-1][0] and not self.consumed:
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1306 self.read()
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1307 if not 0 <= newpos <= self._chunkindex[-1][0]:
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1308 raise ValueError('Offset out of range')
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1309
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1310 if self._pos != newpos:
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1311 chunk, internaloffset = self._findchunk(newpos)
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1312 self._payloadstream = util.chunkbuffer(self._payloadchunks(chunk))
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1313 adjust = self.read(internaloffset)
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1314 if len(adjust) != internaloffset:
27227 56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 27198 diff changeset	1315 raise error.Abort(_('Seek failed\n'))
24049 f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1316 self._pos = newpos
f0b498cfc5c8 bundle2.unbundlepart: implement seek() Eric Sumner <ericsumner@fb.com> parents: 24048 diff changeset	1317
32668 a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1318 def _seekfp(self, offset, whence=0):
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1319 """move the underlying file pointer
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1320
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1321 This method is meant for internal usage by the bundle2 protocol only.
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1322 They directly manipulate the low level stream including bundle2 level
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1323 instruction.
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1324
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1325 Do not use it to implement higher-level logic or methods."""
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1326 if self._seekable:
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1327 return self._fp.seek(offset, whence)
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1328 else:
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1329 raise NotImplementedError(_('File pointer is not seekable'))
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1330
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1331 def _tellfp(self):
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1332 """return the file offset, or None if file is not seekable
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1333
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1334 This method is meant for internal usage by the bundle2 protocol only.
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1335 They directly manipulate the low level stream including bundle2 level
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1336 instruction.
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1337
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1338 Do not use it to implement higher-level logic or methods."""
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1339 if self._seekable:
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1340 try:
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1341 return self._fp.tell()
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1342 except IOError as e:
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1343 if e.errno == errno.ESPIPE:
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1344 self._seekable = False
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1345 else:
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1346 raise
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1347 return None
a02e773008f5 bundle2: move 'seek' and 'tell' methods off the unpackermixin class Pierre-Yves David <pierre-yves.david@ens-lyon.org> parents: 32642 diff changeset	1348
25801 5a5b7046d00e bundle2: add an informative comment to the capability dict Pierre-Yves David <pierre-yves.david@fb.com> parents: 25799 diff changeset	1349 # These are only the static capabilities.
5a5b7046d00e bundle2: add an informative comment to the capability dict Pierre-Yves David <pierre-yves.david@fb.com> parents: 25799 diff changeset	1350 # Check the 'getrepocaps' function for the rest.
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1351 capabilities = {'HG20': (),
26029 d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 26028 diff changeset	1352 'error': ('abort', 'unsupportedcontent', 'pushraced',
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 26028 diff changeset	1353 'pushkey'),
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1354 'listkeys': (),
e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1355 'pushkey': (),
23029 149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1356 'digests': tuple(sorted(util.DIGESTS.keys())),
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1357 'remote-changegroup': ('http', 'https'),
25935 d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1358 'hgtagsfnodes': (),
22341 2d16b39601b5 obsmarker: move bundle2caps from the localrepo class to the bundle2 module Pierre-Yves David <pierre-yves.david@fb.com> parents: 22340 diff changeset	1359 }
2d16b39601b5 obsmarker: move bundle2caps from the localrepo class to the bundle2 module Pierre-Yves David <pierre-yves.david@fb.com> parents: 22340 diff changeset	1360
23439 743736fc7c41 bundle2-push: provide transaction to reply unbundler Eric Sumner <ericsumner@fb.com> parents: 23438 diff changeset	1361 def getrepocaps(repo, allowpushback=False):
22342 262c5cc126c1 bundle2: introduce a `getrepocaps` to retrieve the bundle2 caps of a repo Pierre-Yves David <pierre-yves.david@fb.com> parents: 22341 diff changeset	1362 """return the bundle2 capabilities for a given repo
262c5cc126c1 bundle2: introduce a `getrepocaps` to retrieve the bundle2 caps of a repo Pierre-Yves David <pierre-yves.david@fb.com> parents: 22341 diff changeset	1363
22343 76677a2c1cfd bundle2: advertise the obsmarker part in bundle2 capabilities Pierre-Yves David <pierre-yves.david@fb.com> parents: 22342 diff changeset	1364 Exists to allow extensions (like evolution) to mutate the capabilities.
22342 262c5cc126c1 bundle2: introduce a `getrepocaps` to retrieve the bundle2 caps of a repo Pierre-Yves David <pierre-yves.david@fb.com> parents: 22341 diff changeset	1365 """
22343 76677a2c1cfd bundle2: advertise the obsmarker part in bundle2 capabilities Pierre-Yves David <pierre-yves.david@fb.com> parents: 22342 diff changeset	1366 caps = capabilities.copy()
28624 88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066) Martin von Zweigbergk <martinvonz@google.com> parents: 28534 diff changeset	1367 caps['changegroup'] = tuple(sorted(
88609cfa3745 changegroup: fix pulling to treemanifest repo from flat repo (issue5066) Martin von Zweigbergk <martinvonz@google.com> parents: 28534 diff changeset	1368 changegroup.supportedincomingversions(repo)))
22953 b1d694d3975e obsolete: add exchange option Durham Goode <durham@fb.com> parents: 22661 diff changeset	1369 if obsolete.isenabled(repo, obsolete.exchangeopt):
22343 76677a2c1cfd bundle2: advertise the obsmarker part in bundle2 capabilities Pierre-Yves David <pierre-yves.david@fb.com> parents: 22342 diff changeset	1370 supportedformat = tuple('V%i' % v for v in obsolete.formats)
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1371 caps['obsmarkers'] = supportedformat
23439 743736fc7c41 bundle2-push: provide transaction to reply unbundler Eric Sumner <ericsumner@fb.com> parents: 23438 diff changeset	1372 if allowpushback:
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1373 caps['pushback'] = ()
34005 593ad8df9dd2 configitems: register the 'server.concurrent-push-mode' config Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33942 diff changeset	1374 cpmode = repo.ui.config('server', 'concurrent-push-mode')
33675 a7851519ea02 check-concurrency: expose the feature as 'concurrent-push-mode' Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33492 diff changeset	1375 if cpmode == 'check-related':
33492 16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1376 caps['checkheads'] = ('related',)
22343 76677a2c1cfd bundle2: advertise the obsmarker part in bundle2 capabilities Pierre-Yves David <pierre-yves.david@fb.com> parents: 22342 diff changeset	1377 return caps
22342 262c5cc126c1 bundle2: introduce a `getrepocaps` to retrieve the bundle2 caps of a repo Pierre-Yves David <pierre-yves.david@fb.com> parents: 22341 diff changeset	1378
21644 17755dd8c509 bundle2: introduce a bundle2caps function Pierre-Yves David <pierre-yves.david@fb.com> parents: 21628 diff changeset	1379 def bundle2caps(remote):
23139 e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues Mads Kiilerich <madski@unity3d.com> parents: 23067 diff changeset	1380 """return the bundle capabilities of a peer as dict"""
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1381 raw = remote.capable('bundle2')
21644 17755dd8c509 bundle2: introduce a bundle2caps function Pierre-Yves David <pierre-yves.david@fb.com> parents: 21628 diff changeset	1382 if not raw and raw != '':
17755dd8c509 bundle2: introduce a bundle2caps function Pierre-Yves David <pierre-yves.david@fb.com> parents: 21628 diff changeset	1383 return {}
29633 032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat timeless <timeless@mozdev.org> parents: 29422 diff changeset	1384 capsblob = urlreq.unquote(remote.capable('bundle2'))
21644 17755dd8c509 bundle2: introduce a bundle2caps function Pierre-Yves David <pierre-yves.david@fb.com> parents: 21628 diff changeset	1385 return decodecaps(capsblob)
21014 a6246bba7b9e bundle2: add an unbundle part responsible from unbundling part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21013 diff changeset	1386
22344 9829b7948100 bundle2: add a `obsmarkersversion` function to extract supported version Pierre-Yves David <pierre-yves.david@fb.com> parents: 22343 diff changeset	1387 def obsmarkersversion(caps):
9829b7948100 bundle2: add a `obsmarkersversion` function to extract supported version Pierre-Yves David <pierre-yves.david@fb.com> parents: 22343 diff changeset	1388 """extract the list of supported obsmarkers versions from a bundle2caps dict
9829b7948100 bundle2: add a `obsmarkersversion` function to extract supported version Pierre-Yves David <pierre-yves.david@fb.com> parents: 22343 diff changeset	1389 """
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1390 obscaps = caps.get('obsmarkers', ())
22344 9829b7948100 bundle2: add a `obsmarkersversion` function to extract supported version Pierre-Yves David <pierre-yves.david@fb.com> parents: 22343 diff changeset	1391 return [int(c[1:]) for c in obscaps if c.startswith('V')]
9829b7948100 bundle2: add a `obsmarkersversion` function to extract supported version Pierre-Yves David <pierre-yves.david@fb.com> parents: 22343 diff changeset	1392
33004 9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1393 def writenewbundle(ui, repo, source, filename, bundletype, outgoing, opts,
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1394 vfs=None, compression=None, compopts=None):
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1395 if bundletype.startswith('HG10'):
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1396 cg = changegroup.getchangegroup(repo, source, outgoing, version='01')
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1397 return writebundle(ui, cg, filename, bundletype, vfs=vfs,
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1398 compression=compression, compopts=compopts)
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1399 elif not bundletype.startswith('HG20'):
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1400 raise error.ProgrammingError('unknown bundle type: %s' % bundletype)
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1401
33299 37d70ba1d9d1 bundle: add an experimental knob to include obsmarkers in bundle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33298 diff changeset	1402 caps = {}
37d70ba1d9d1 bundle: add an experimental knob to include obsmarkers in bundle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33298 diff changeset	1403 if 'obsolescence' in opts:
37d70ba1d9d1 bundle: add an experimental knob to include obsmarkers in bundle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33298 diff changeset	1404 caps['obsmarkers'] = ('V1',)
37d70ba1d9d1 bundle: add an experimental knob to include obsmarkers in bundle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33298 diff changeset	1405 bundle = bundle20(ui, caps)
33004 9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1406 bundle.setcompression(compression, compopts)
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1407 _addpartsfromopts(ui, repo, bundle, source, outgoing, opts)
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1408 chunkiter = bundle.getchunks()
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1409
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1410 return changegroup.writechunks(ui, chunkiter, filename, vfs=vfs)
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1411
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1412 def _addpartsfromopts(ui, repo, bundler, source, outgoing, opts):
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1413 # We should eventually reconcile this logic with the one behind
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1414 # 'exchange.getbundle2partsgenerator'.
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1415 #
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1416 # The type of input from 'getbundle' and 'writenewbundle' are a bit
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1417 # different right now. So we keep them separated for now for the sake of
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1418 # simplicity.
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1419
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1420 # we always want a changegroup in such bundle
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1421 cgversion = opts.get('cg.version')
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1422 if cgversion is None:
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1423 cgversion = changegroup.safeversion(repo)
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1424 cg = changegroup.getchangegroup(repo, source, outgoing,
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1425 version=cgversion)
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1426 part = bundler.newpart('changegroup', data=cg.getchunks())
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1427 part.addparam('version', cg.version)
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1428 if 'clcount' in cg.extras:
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1429 part.addparam('nbchanges', str(cg.extras['clcount']),
9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1430 mandatory=False)
34191 e3867c712d51 bundle2: automatically add 'targetphase' parameter in writenewbundle Boris Feld <boris.feld@octobus.net> parents: 34190 diff changeset	1431 if opts.get('phases') and repo.revs('%ln and secret()',
e3867c712d51 bundle2: automatically add 'targetphase' parameter in writenewbundle Boris Feld <boris.feld@octobus.net> parents: 34190 diff changeset	1432 outgoing.missingheads):
e3867c712d51 bundle2: automatically add 'targetphase' parameter in writenewbundle Boris Feld <boris.feld@octobus.net> parents: 34190 diff changeset	1433 part.addparam('targetphase', '%d' % phases.secret, mandatory=False)
33004 9dc36df78403 bundle: introduce an higher level function to write bundle on disk Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32969 diff changeset	1434
33006 d7f93ebbbbdf bundle: add optional 'tagsfnodecache' data to on disk bundle (issue5543) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33005 diff changeset	1435 addparttagsfnodescache(repo, bundler, outgoing)
d7f93ebbbbdf bundle: add optional 'tagsfnodecache' data to on disk bundle (issue5543) Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33005 diff changeset	1436
33299 37d70ba1d9d1 bundle: add an experimental knob to include obsmarkers in bundle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33298 diff changeset	1437 if opts.get('obsolescence', False):
37d70ba1d9d1 bundle: add an experimental knob to include obsmarkers in bundle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33298 diff changeset	1438 obsmarkers = repo.obsstore.relevantmarkers(outgoing.missing)
37d70ba1d9d1 bundle: add an experimental knob to include obsmarkers in bundle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33298 diff changeset	1439 buildobsmarkerspart(bundler, obsmarkers)
37d70ba1d9d1 bundle: add an experimental knob to include obsmarkers in bundle Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33298 diff changeset	1440
33814 e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1441 if opts.get('phases', False):
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1442 headsbyphase = phases.subsetphaseheads(repo, outgoing.missing)
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1443 phasedata = []
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1444 for phase in phases.allphases:
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1445 for head in headsbyphase[phase]:
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1446 phasedata.append(_pack(_fphasesentry, phase, head))
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1447 bundler.newpart('phase-heads', data=''.join(phasedata))
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1448
33005 6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1449 def addparttagsfnodescache(repo, bundler, outgoing):
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1450 # we include the tags fnode cache for the bundle changeset
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1451 # (as an optional parts)
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1452 cache = tags.hgtagsfnodescache(repo.unfiltered())
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1453 chunks = []
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1454
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1455 # .hgtags fnodes are only relevant for head changesets. While we could
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1456 # transfer values for all known nodes, there will likely be little to
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1457 # no benefit.
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1458 #
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1459 # We don't bother using a generator to produce output data because
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1460 # a) we only have 40 bytes per head and even esoteric numbers of heads
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1461 # consume little memory (1M heads is 40MB) b) we don't want to send the
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1462 # part if we don't have entries and knowing if we have entries requires
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1463 # cache lookups.
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1464 for node in outgoing.missingheads:
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1465 # Don't compute missing, as this may slow down serving.
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1466 fnode = cache.getfnode(node, computemissing=False)
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1467 if fnode is not None:
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1468 chunks.extend([node, fnode])
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1469
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1470 if chunks:
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1471 bundler.newpart('hgtagsfnodes', data=''.join(chunks))
6068712cbf03 bundle2: move tagsfnodecache generation in a generic function Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33004 diff changeset	1472
33298 e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33170 diff changeset	1473 def buildobsmarkerspart(bundler, markers):
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33170 diff changeset	1474 """add an obsmarker part to the bundler with <markers>
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33170 diff changeset	1475
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33170 diff changeset	1476 No part is created if markers is empty.
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33170 diff changeset	1477 Raises ValueError if the bundler doesn't support any known obsmarker format.
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33170 diff changeset	1478 """
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33170 diff changeset	1479 if not markers:
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33170 diff changeset	1480 return None
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33170 diff changeset	1481
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33170 diff changeset	1482 remoteversions = obsmarkersversion(bundler.capabilities)
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33170 diff changeset	1483 version = obsolete.commonversion(remoteversions)
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33170 diff changeset	1484 if version is None:
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33170 diff changeset	1485 raise ValueError('bundler does not support common obsmarker format')
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33170 diff changeset	1486 stream = obsolete.encodemarkers(markers, True, version=version)
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33170 diff changeset	1487 return bundler.newpart('obsmarkers', data=stream)
e70d6dbde713 bundle2: move function building obsmarker-part in the bundle2 module Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33170 diff changeset	1488
31536 511a4bf52754 bundle2: allow compression options to be passed to compressor Gregory Szorc <gregory.szorc@gmail.com> parents: 31218 diff changeset	1489 def writebundle(ui, cg, filename, bundletype, vfs=None, compression=None,
511a4bf52754 bundle2: allow compression options to be passed to compressor Gregory Szorc <gregory.szorc@gmail.com> parents: 31218 diff changeset	1490 compopts=None):
29407 ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1491 """Write a bundle file and return its filename.
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1492
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1493 Existing files will not be overwritten.
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1494 If no filename is specified, a temporary file is created.
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1495 bz2 compression can be turned off.
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1496 The bundle file will be deleted in case of errors.
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1497 """
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1498
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1499 if bundletype == "HG20":
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1500 bundle = bundle20(ui)
31536 511a4bf52754 bundle2: allow compression options to be passed to compressor Gregory Szorc <gregory.szorc@gmail.com> parents: 31218 diff changeset	1501 bundle.setcompression(compression, compopts)
29407 ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1502 part = bundle.newpart('changegroup', data=cg.getchunks())
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1503 part.addparam('version', cg.version)
30354 953839de96ab bundle2: store changeset count when creating file bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 30352 diff changeset	1504 if 'clcount' in cg.extras:
953839de96ab bundle2: store changeset count when creating file bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 30352 diff changeset	1505 part.addparam('nbchanges', str(cg.extras['clcount']),
953839de96ab bundle2: store changeset count when creating file bundles Gregory Szorc <gregory.szorc@gmail.com> parents: 30352 diff changeset	1506 mandatory=False)
29407 ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1507 chunkiter = bundle.getchunks()
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1508 else:
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1509 # compression argument is only for the bundle2 case
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1510 assert compression is None
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1511 if cg.version != '01':
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1512 raise error.Abort(_('old bundle types only supports v1 '
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1513 'changegroups'))
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1514 header, comp = bundletypes[bundletype]
31130 f81002f736d7 bundle2: use new compression engine API for compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30956 diff changeset	1515 if comp not in util.compengines.supportedbundletypes:
29407 ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1516 raise error.Abort(_('unknown stream compression type: %s')
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1517 % comp)
31130 f81002f736d7 bundle2: use new compression engine API for compression Gregory Szorc <gregory.szorc@gmail.com> parents: 30956 diff changeset	1518 compengine = util.compengines.forbundletype(comp)
29407 ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1519 def chunkiter():
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1520 yield header
31536 511a4bf52754 bundle2: allow compression options to be passed to compressor Gregory Szorc <gregory.szorc@gmail.com> parents: 31218 diff changeset	1521 for chunk in compengine.compressstream(cg.getchunks(), compopts):
31136 5925bda42dbd bundle2: use compressstream compression engine API Gregory Szorc <gregory.szorc@gmail.com> parents: 31132 diff changeset	1522 yield chunk
29407 ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1523 chunkiter = chunkiter()
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1524
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1525 # parse the changegroup data, otherwise we will block
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1526 # in case of sshrepo because we don't know the end of the stream
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1527 return changegroup.writechunks(ui, chunkiter, filename, vfs=vfs)
ae53ecc47414 bundle: move writebundle() from changegroup.py to bundle2.py (API) Martin von Zweigbergk <martinvonz@google.com> parents: 28624 diff changeset	1528
33820 d765ad56081f bundle: make combinechangegroupresults() take a bundleoperation Martin von Zweigbergk <martinvonz@google.com> parents: 33819 diff changeset	1529 def combinechangegroupresults(op):
33819 52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1530 """logic to combine 0 or more addchangegroup results into one"""
33820 d765ad56081f bundle: make combinechangegroupresults() take a bundleoperation Martin von Zweigbergk <martinvonz@google.com> parents: 33819 diff changeset	1531 results = [r.get('return', 0)
d765ad56081f bundle: make combinechangegroupresults() take a bundleoperation Martin von Zweigbergk <martinvonz@google.com> parents: 33819 diff changeset	1532 for r in op.records['changegroup']]
33819 52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1533 changedheads = 0
52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1534 result = 1
52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1535 for ret in results:
52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1536 # If any changegroup result is 0, return 0
52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1537 if ret == 0:
52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1538 result = 0
52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1539 break
52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1540 if ret < -1:
52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1541 changedheads += ret + 1
52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1542 elif ret > 1:
52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1543 changedheads += ret - 1
52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1544 if changedheads > 0:
52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1545 result = 1 + changedheads
52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1546 elif changedheads < 0:
52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1547 result = -1 + changedheads
52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1548 return result
52c7060b707a bundle: move combineresults() from changegroup to bundle2 Martin von Zweigbergk <martinvonz@google.com> parents: 33818 diff changeset	1549
34190 39d4e5a66f5a bundle2: support the 'targetphase' parameter for the changegroup part Boris Feld <boris.feld@octobus.net> parents: 34035 diff changeset	1550 @parthandler('changegroup', ('version', 'nbchanges', 'treemanifest',
39d4e5a66f5a bundle2: support the 'targetphase' parameter for the changegroup part Boris Feld <boris.feld@octobus.net> parents: 34035 diff changeset	1551 'targetphase'))
20998 93a3c5b58635 bundle2: use reply part to return result of addchangegroup Pierre-Yves David <pierre-yves.david@fb.com> parents: 20997 diff changeset	1552 def handlechangegroup(op, inpart):
20950 c7ceae0faf69 bundle2: first crude version of bundling changeset with bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 20949 diff changeset	1553 """apply a changegroup part on the repo
c7ceae0faf69 bundle2: first crude version of bundling changeset with bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 20949 diff changeset	1554
c7ceae0faf69 bundle2: first crude version of bundling changeset with bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 20949 diff changeset	1555 This is a very early implementation that will massive rework before being
c7ceae0faf69 bundle2: first crude version of bundling changeset with bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 20949 diff changeset	1556 inflicted to any end-user.
c7ceae0faf69 bundle2: first crude version of bundling changeset with bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 20949 diff changeset	1557 """
33713 af31d531dda0 changegroup: let callers pass in transaction to apply() (API) Martin von Zweigbergk <martinvonz@google.com> parents: 33675 diff changeset	1558 tr = op.gettransaction()
23170 02e8f9b60052 bundle2: support a "version" argument in `changegroup` part Pierre-Yves David <pierre-yves.david@fb.com> parents: 23169 diff changeset	1559 unpackerversion = inpart.params.get('version', '01')
02e8f9b60052 bundle2: support a "version" argument in `changegroup` part Pierre-Yves David <pierre-yves.david@fb.com> parents: 23169 diff changeset	1560 # We should raise an appropriate exception here
28406 a40e2f7fe49d changegroup: hide packermap behind methods Martin von Zweigbergk <martinvonz@google.com> parents: 28389 diff changeset	1561 cg = changegroup.getunbundler(unpackerversion, inpart, None)
23001 4df9b5e62f70 bundle2: add a comment about addchangegroup source and url Pierre-Yves David <pierre-yves.david@fb.com> parents: 22953 diff changeset	1562 # the source and url passed here are overwritten by the one contained in
4df9b5e62f70 bundle2: add a comment about addchangegroup source and url Pierre-Yves David <pierre-yves.david@fb.com> parents: 22953 diff changeset	1563 # the transaction.hookargs argument. So 'bundle2' is a placeholder
26061 ca656f3dffd7 bundle2: provide number of changesets information to 'addchangegroup' Pierre-Yves David <pierre-yves.david@fb.com> parents: 26043 diff changeset	1564 nbchangesets = None
ca656f3dffd7 bundle2: provide number of changesets information to 'addchangegroup' Pierre-Yves David <pierre-yves.david@fb.com> parents: 26043 diff changeset	1565 if 'nbchanges' in inpart.params:
ca656f3dffd7 bundle2: provide number of changesets information to 'addchangegroup' Pierre-Yves David <pierre-yves.david@fb.com> parents: 26043 diff changeset	1566 nbchangesets = int(inpart.params.get('nbchanges'))
28389 5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest Martin von Zweigbergk <martinvonz@google.com> parents: 27469 diff changeset	1567 if ('treemanifest' in inpart.params and
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest Martin von Zweigbergk <martinvonz@google.com> parents: 27469 diff changeset	1568 'treemanifest' not in op.repo.requirements):
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest Martin von Zweigbergk <martinvonz@google.com> parents: 27469 diff changeset	1569 if len(op.repo.changelog) != 0:
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest Martin von Zweigbergk <martinvonz@google.com> parents: 27469 diff changeset	1570 raise error.Abort(_(
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest Martin von Zweigbergk <martinvonz@google.com> parents: 27469 diff changeset	1571 "bundle contains tree manifests, but local repo is "
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest Martin von Zweigbergk <martinvonz@google.com> parents: 27469 diff changeset	1572 "non-empty and does not use tree manifests"))
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest Martin von Zweigbergk <martinvonz@google.com> parents: 27469 diff changeset	1573 op.repo.requirements.add('treemanifest')
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest Martin von Zweigbergk <martinvonz@google.com> parents: 27469 diff changeset	1574 op.repo._applyopenerreqs()
5c0fd878779c treemanifests: set bundle2 part parameter indicating treemanifest Martin von Zweigbergk <martinvonz@google.com> parents: 27469 diff changeset	1575 op.repo._writerequirements()
34190 39d4e5a66f5a bundle2: support the 'targetphase' parameter for the changegroup part Boris Feld <boris.feld@octobus.net> parents: 34035 diff changeset	1576 extrakwargs = {}
39d4e5a66f5a bundle2: support the 'targetphase' parameter for the changegroup part Boris Feld <boris.feld@octobus.net> parents: 34035 diff changeset	1577 targetphase = inpart.params.get('targetphase')
39d4e5a66f5a bundle2: support the 'targetphase' parameter for the changegroup part Boris Feld <boris.feld@octobus.net> parents: 34035 diff changeset	1578 if targetphase is not None:
39d4e5a66f5a bundle2: support the 'targetphase' parameter for the changegroup part Boris Feld <boris.feld@octobus.net> parents: 34035 diff changeset	1579 extrakwargs['targetphase'] = int(targetphase)
33821 f0efd2bffe1e bundle: extract _processchangegroup() method Martin von Zweigbergk <martinvonz@google.com> parents: 33820 diff changeset	1580 ret = _processchangegroup(op, cg, tr, 'bundle2', 'bundle2',
34190 39d4e5a66f5a bundle2: support the 'targetphase' parameter for the changegroup part Boris Feld <boris.feld@octobus.net> parents: 34035 diff changeset	1581 expectedtotal=nbchangesets, **extrakwargs)
20998 93a3c5b58635 bundle2: use reply part to return result of addchangegroup Pierre-Yves David <pierre-yves.david@fb.com> parents: 20997 diff changeset	1582 if op.reply is not None:
23139 e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues Mads Kiilerich <madski@unity3d.com> parents: 23067 diff changeset	1583 # This is definitely not the final form of this
20998 93a3c5b58635 bundle2: use reply part to return result of addchangegroup Pierre-Yves David <pierre-yves.david@fb.com> parents: 20997 diff changeset	1584 # return. But one need to start somewhere.
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1585 part = op.reply.newpart('reply:changegroup', mandatory=False)
34458 24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes Augie Fackler <augie@google.com> parents: 34457 diff changeset	1586 part.addparam(
24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes Augie Fackler <augie@google.com> parents: 34457 diff changeset	1587 'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False)
21606 e55888447958 bundle2: update part creators to ``addparam`` when relevant Pierre-Yves David <pierre-yves.david@fb.com> parents: 21605 diff changeset	1588 part.addparam('return', '%i' % ret, mandatory=False)
21019 3dc09f831a2e bundle2: lazy unbundle of part payload Pierre-Yves David <pierre-yves.david@fb.com> parents: 21016 diff changeset	1589 assert not inpart.read()
20950 c7ceae0faf69 bundle2: first crude version of bundling changeset with bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 20949 diff changeset	1590
23029 149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1591 _remotechangegroupparams = tuple(['url', 'size', 'digests'] +
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1592 ['digest:%s' % k for k in util.DIGESTS.keys()])
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1593 @parthandler('remote-changegroup', _remotechangegroupparams)
23029 149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1594 def handleremotechangegroup(op, inpart):
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1595 """apply a bundle10 on the repo, given an url and validation information
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1596
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1597 All the information about the remote bundle to import are given as
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1598 parameters. The parameters include:
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1599 - url: the url to the bundle10.
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1600 - size: the bundle10 file size. It is used to validate what was
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1601 retrieved by the client matches the server knowledge about the bundle.
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1602 - digests: a space separated list of the digest types provided as
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1603 parameters.
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1604 - digest:<digest-type>: the hexadecimal representation of the digest with
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1605 that name. Like the size, it is used to validate what was retrieved by
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1606 the client matches what the server knows about the bundle.
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1607
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1608 When multiple digest types are given, all of them are checked.
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1609 """
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1610 try:
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1611 raw_url = inpart.params['url']
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1612 except KeyError:
27227 56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 27198 diff changeset	1613 raise error.Abort(_('remote-changegroup: missing "%s" param') % 'url')
23029 149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1614 parsed_url = util.url(raw_url)
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1615 if parsed_url.scheme not in capabilities['remote-changegroup']:
27227 56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 27198 diff changeset	1616 raise error.Abort(_('remote-changegroup does not support %s urls') %
23029 149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1617 parsed_url.scheme)
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1618
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1619 try:
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1620 size = int(inpart.params['size'])
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1621 except ValueError:
27227 56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 27198 diff changeset	1622 raise error.Abort(_('remote-changegroup: invalid value for param "%s"')
23029 149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1623 % 'size')
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1624 except KeyError:
27227 56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 27198 diff changeset	1625 raise error.Abort(_('remote-changegroup: missing "%s" param') % 'size')
23029 149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1626
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1627 digests = {}
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1628 for typ in inpart.params.get('digests', '').split():
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1629 param = 'digest:%s' % typ
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1630 try:
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1631 value = inpart.params[param]
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1632 except KeyError:
27227 56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 27198 diff changeset	1633 raise error.Abort(_('remote-changegroup: missing "%s" param') %
23029 149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1634 param)
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1635 digests[typ] = value
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1636
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1637 real_part = util.digestchecker(url.open(op.ui, raw_url), size, digests)
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1638
33713 af31d531dda0 changegroup: let callers pass in transaction to apply() (API) Martin von Zweigbergk <martinvonz@google.com> parents: 33675 diff changeset	1639 tr = op.gettransaction()
26503 8221fefaea08 bundle2: use absolute_import Gregory Szorc <gregory.szorc@gmail.com> parents: 26214 diff changeset	1640 from . import exchange
23029 149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1641 cg = exchange.readbundle(op.repo.ui, real_part, raw_url)
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1642 if not isinstance(cg, changegroup.cg1unpacker):
27227 56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 27198 diff changeset	1643 raise error.Abort(_('%s: not a bundle version 1.0') %
23029 149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1644 util.hidepassword(raw_url))
33821 f0efd2bffe1e bundle: extract _processchangegroup() method Martin von Zweigbergk <martinvonz@google.com> parents: 33820 diff changeset	1645 ret = _processchangegroup(op, cg, tr, 'bundle2', 'bundle2')
23029 149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1646 if op.reply is not None:
23139 e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues Mads Kiilerich <madski@unity3d.com> parents: 23067 diff changeset	1647 # This is definitely not the final form of this
23029 149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1648 # return. But one need to start somewhere.
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1649 part = op.reply.newpart('reply:changegroup')
34458 24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes Augie Fackler <augie@google.com> parents: 34457 diff changeset	1650 part.addparam(
24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes Augie Fackler <augie@google.com> parents: 34457 diff changeset	1651 'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False)
23029 149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1652 part.addparam('return', '%i' % ret, mandatory=False)
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1653 try:
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1654 real_part.validate()
27227 56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 27198 diff changeset	1655 except error.Abort as e:
56b2bcea2529 error: get Abort from 'error' instead of 'util' Pierre-Yves David <pierre-yves.david@fb.com> parents: 27198 diff changeset	1656 raise error.Abort(_('bundle at %s is corrupted:\n%s') %
23029 149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1657 (util.hidepassword(raw_url), str(e)))
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1658 assert not inpart.read()
149fc8a44184 bundle2: client side support for a part to import external bundles Mike Hommey <mh@glandium.org> parents: 23011 diff changeset	1659
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1660 @parthandler('reply:changegroup', ('return', 'in-reply-to'))
22548 8a1ae133770a bundle2: rename functions that have the same name Mike Hommey <mh@glandium.org> parents: 22390 diff changeset	1661 def handlereplychangegroup(op, inpart):
21611 71b7b3f79a3c bundle2: use the new ``part.params`` dictionary Pierre-Yves David <pierre-yves.david@fb.com> parents: 21610 diff changeset	1662 ret = int(inpart.params['return'])
71b7b3f79a3c bundle2: use the new ``part.params`` dictionary Pierre-Yves David <pierre-yves.david@fb.com> parents: 21610 diff changeset	1663 replyto = int(inpart.params['in-reply-to'])
71b7b3f79a3c bundle2: use the new ``part.params`` dictionary Pierre-Yves David <pierre-yves.david@fb.com> parents: 21610 diff changeset	1664 op.records.add('changegroup', {'return': ret}, replyto)
20950 c7ceae0faf69 bundle2: first crude version of bundling changeset with bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 20949 diff changeset	1665
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1666 @parthandler('check:heads')
22548 8a1ae133770a bundle2: rename functions that have the same name Mike Hommey <mh@glandium.org> parents: 22390 diff changeset	1667 def handlecheckheads(op, inpart):
21060 0bea9db7543b bundle2: add a "check:heads" handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21024 diff changeset	1668 """check that head of the repo did not change
0bea9db7543b bundle2: add a "check:heads" handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21024 diff changeset	1669
0bea9db7543b bundle2: add a "check:heads" handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21024 diff changeset	1670 This is used to detect a push race when using unbundle.
0bea9db7543b bundle2: add a "check:heads" handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21024 diff changeset	1671 This replaces the "heads" argument of unbundle."""
0bea9db7543b bundle2: add a "check:heads" handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21024 diff changeset	1672 h = inpart.read(20)
0bea9db7543b bundle2: add a "check:heads" handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21024 diff changeset	1673 heads = []
0bea9db7543b bundle2: add a "check:heads" handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21024 diff changeset	1674 while len(h) == 20:
0bea9db7543b bundle2: add a "check:heads" handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21024 diff changeset	1675 heads.append(h)
0bea9db7543b bundle2: add a "check:heads" handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21024 diff changeset	1676 h = inpart.read(20)
0bea9db7543b bundle2: add a "check:heads" handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21024 diff changeset	1677 assert not h
27198 ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock Durham Goode <durham@fb.com> parents: 27175 diff changeset	1678 # Trigger a transaction so that we are guaranteed to have the lock now.
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock Durham Goode <durham@fb.com> parents: 27175 diff changeset	1679 if op.ui.configbool('experimental', 'bundle2lazylocking'):
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock Durham Goode <durham@fb.com> parents: 27175 diff changeset	1680 op.gettransaction()
30044 077d0535f51f bundle2: don't assume ordering of heads checked after push Mads Kiilerich <madski@unity3d.com> parents: 29633 diff changeset	1681 if sorted(heads) != sorted(op.repo.heads()):
21185 5b3717e1a3ea bundle2: add an error message to push race error Pierre-Yves David <pierre-yves.david@fb.com> parents: 21184 diff changeset	1682 raise error.PushRaced('repository changed while pushing - '
5b3717e1a3ea bundle2: add an error message to push race error Pierre-Yves David <pierre-yves.david@fb.com> parents: 21184 diff changeset	1683 'please try again')
21130 1ff06386217f bundle2: introduce `replycaps` part for on-demand reply Pierre-Yves David <pierre-yves.david@fb.com> parents: 21129 diff changeset	1684
33492 16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1685 @parthandler('check:updated-heads')
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1686 def handlecheckupdatedheads(op, inpart):
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1687 """check for race on the heads touched by a push
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1688
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1689 This is similar to 'check:heads' but focus on the heads actually updated
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1690 during the push. If other activities happen on unrelated heads, it is
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1691 ignored.
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1692
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1693 This allow server with high traffic to avoid push contention as long as
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1694 unrelated parts of the graph are involved."""
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1695 h = inpart.read(20)
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1696 heads = []
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1697 while len(h) == 20:
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1698 heads.append(h)
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1699 h = inpart.read(20)
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1700 assert not h
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1701 # trigger a transaction so that we are guaranteed to have the lock now.
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1702 if op.ui.configbool('experimental', 'bundle2lazylocking'):
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1703 op.gettransaction()
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1704
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1705 currentheads = set()
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1706 for ls in op.repo.branchmap().itervalues():
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1707 currentheads.update(ls)
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1708
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1709 for h in heads:
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1710 if h not in currentheads:
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1711 raise error.PushRaced('repository changed while pushing - '
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1712 'please try again')
16ada4cbb1a9 push: add a way to allow concurrent pushes on unrelated heads Pierre-Yves David <pierre-yves.david@octobus.net> parents: 33299 diff changeset	1713
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1714 @parthandler('output')
21131 b7435117d951 bundle2: capture remote stdout while unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 21130 diff changeset	1715 def handleoutput(op, inpart):
b7435117d951 bundle2: capture remote stdout while unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 21130 diff changeset	1716 """forward output captured on the server to the client"""
b7435117d951 bundle2: capture remote stdout while unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 21130 diff changeset	1717 for line in inpart.read().splitlines():
30621 ccd436f7db6d bundle2: localize handleoutput remote prompts Akihiko Odaki <akihiko.odaki.4i@stu.hosei.ac.jp> parents: 30617 diff changeset	1718 op.ui.status(_('remote: %s\n') % line)
21131 b7435117d951 bundle2: capture remote stdout while unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 21130 diff changeset	1719
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1720 @parthandler('replycaps')
21130 1ff06386217f bundle2: introduce `replycaps` part for on-demand reply Pierre-Yves David <pierre-yves.david@fb.com> parents: 21129 diff changeset	1721 def handlereplycaps(op, inpart):
1ff06386217f bundle2: introduce `replycaps` part for on-demand reply Pierre-Yves David <pierre-yves.david@fb.com> parents: 21129 diff changeset	1722 """Notify that a reply bundle should be created
1ff06386217f bundle2: introduce `replycaps` part for on-demand reply Pierre-Yves David <pierre-yves.david@fb.com> parents: 21129 diff changeset	1723
21138 f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	1724 The payload contains the capabilities information for the reply"""
f469879d27ec bundle2: extract capabilities decoding Pierre-Yves David <pierre-yves.david@fb.com> parents: 21137 diff changeset	1725 caps = decodecaps(inpart.read())
21130 1ff06386217f bundle2: introduce `replycaps` part for on-demand reply Pierre-Yves David <pierre-yves.david@fb.com> parents: 21129 diff changeset	1726 if op.reply is None:
21135 98fbf3adfd83 bundle2: add capabilities support in `replycaps` part Pierre-Yves David <pierre-yves.david@fb.com> parents: 21134 diff changeset	1727 op.reply = bundle20(op.ui, caps)
21131 b7435117d951 bundle2: capture remote stdout while unbundling Pierre-Yves David <pierre-yves.david@fb.com> parents: 21130 diff changeset	1728
27469 58f1645f72c3 bundle2: attribute remote failures to remote (issue4788) Gregory Szorc <gregory.szorc@gmail.com> parents: 27442 diff changeset	1729 class AbortFromPart(error.Abort):
58f1645f72c3 bundle2: attribute remote failures to remote (issue4788) Gregory Szorc <gregory.szorc@gmail.com> parents: 27442 diff changeset	1730 """Sub-class of Abort that denotes an error from a bundle2 part."""
58f1645f72c3 bundle2: attribute remote failures to remote (issue4788) Gregory Szorc <gregory.szorc@gmail.com> parents: 27442 diff changeset	1731
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1732 @parthandler('error:abort', ('message', 'hint'))
24920 bb67e52362d6 bundle2: fix names for error part handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 24912 diff changeset	1733 def handleerrorabort(op, inpart):
21177 952af771bc17 bundle2: gracefully handle abort during unbundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 21176 diff changeset	1734 """Used to transmit abort error over the wire"""
27469 58f1645f72c3 bundle2: attribute remote failures to remote (issue4788) Gregory Szorc <gregory.szorc@gmail.com> parents: 27442 diff changeset	1735 raise AbortFromPart(inpart.params['message'],
58f1645f72c3 bundle2: attribute remote failures to remote (issue4788) Gregory Szorc <gregory.szorc@gmail.com> parents: 27442 diff changeset	1736 hint=inpart.params.get('hint'))
21183 4345274adc4b bundle2: gracefully handle UnknownPartError during unbundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 21179 diff changeset	1737
26029 d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 26028 diff changeset	1738 @parthandler('error:pushkey', ('namespace', 'key', 'new', 'old', 'ret',
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 26028 diff changeset	1739 'in-reply-to'))
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 26028 diff changeset	1740 def handleerrorpushkey(op, inpart):
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 26028 diff changeset	1741 """Used to transmit failure of a mandatory pushkey over the wire"""
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 26028 diff changeset	1742 kwargs = {}
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 26028 diff changeset	1743 for name in ('namespace', 'key', 'new', 'old', 'ret'):
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 26028 diff changeset	1744 value = inpart.params.get(name)
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 26028 diff changeset	1745 if value is not None:
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 26028 diff changeset	1746 kwargs[name] = value
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 26028 diff changeset	1747 raise error.PushkeyFailed(inpart.params['in-reply-to'], **kwargs)
d8e7b0781ad7 bundle2: convey PushkeyFailed error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 26028 diff changeset	1748
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1749 @parthandler('error:unsupportedcontent', ('parttype', 'params'))
24920 bb67e52362d6 bundle2: fix names for error part handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 24912 diff changeset	1750 def handleerrorunsupportedcontent(op, inpart):
21619 292331e906d7 bundle2: rename b2x:error:unknownpart to b2x:error:unsupportedcontent Pierre-Yves David <pierre-yves.david@fb.com> parents: 21618 diff changeset	1751 """Used to transmit unknown content error over the wire"""
21622 457492741007 bundle2: support transmission of params error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 21620 diff changeset	1752 kwargs = {}
21627 3e8bcc90f07c bundle2: support None parttype in BundleValueError Pierre-Yves David <pierre-yves.david@fb.com> parents: 21626 diff changeset	1753 parttype = inpart.params.get('parttype')
3e8bcc90f07c bundle2: support None parttype in BundleValueError Pierre-Yves David <pierre-yves.david@fb.com> parents: 21626 diff changeset	1754 if parttype is not None:
3e8bcc90f07c bundle2: support None parttype in BundleValueError Pierre-Yves David <pierre-yves.david@fb.com> parents: 21626 diff changeset	1755 kwargs['parttype'] = parttype
21622 457492741007 bundle2: support transmission of params error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 21620 diff changeset	1756 params = inpart.params.get('params')
457492741007 bundle2: support transmission of params error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 21620 diff changeset	1757 if params is not None:
457492741007 bundle2: support transmission of params error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 21620 diff changeset	1758 kwargs['params'] = params.split('\0')
457492741007 bundle2: support transmission of params error over the wire Pierre-Yves David <pierre-yves.david@fb.com> parents: 21620 diff changeset	1759
27019 cff70549a959 bundle2: rename error exception class for unsupported feature Pierre-Yves David <pierre-yves.david@fb.com> parents: 26887 diff changeset	1760 raise error.BundleUnknownFeatureError(**kwargs)
21186 9f3652e851f8 bundle2: gracefully handle PushRaced error during unbundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 21185 diff changeset	1761
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1762 @parthandler('error:pushraced', ('message',))
24920 bb67e52362d6 bundle2: fix names for error part handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 24912 diff changeset	1763 def handleerrorpushraced(op, inpart):
21186 9f3652e851f8 bundle2: gracefully handle PushRaced error during unbundle Pierre-Yves David <pierre-yves.david@fb.com> parents: 21185 diff changeset	1764 """Used to transmit push race error over the wire"""
21611 71b7b3f79a3c bundle2: use the new ``part.params`` dictionary Pierre-Yves David <pierre-yves.david@fb.com> parents: 21610 diff changeset	1765 raise error.ResponseError(_('push failed:'), inpart.params['message'])
21655 35095f332846 bundle: introduce a listkey handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21644 diff changeset	1766
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1767 @parthandler('listkeys', ('namespace',))
21655 35095f332846 bundle: introduce a listkey handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21644 diff changeset	1768 def handlelistkeys(op, inpart):
35095f332846 bundle: introduce a listkey handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21644 diff changeset	1769 """retrieve pushkey namespace content stored in a bundle2"""
35095f332846 bundle: introduce a listkey handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21644 diff changeset	1770 namespace = inpart.params['namespace']
35095f332846 bundle: introduce a listkey handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21644 diff changeset	1771 r = pushkey.decodekeys(inpart.read())
35095f332846 bundle: introduce a listkey handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21644 diff changeset	1772 op.records.add('listkeys', (namespace, r))
21660 e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1773
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1774 @parthandler('pushkey', ('namespace', 'key', 'old', 'new'))
21660 e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1775 def handlepushkey(op, inpart):
e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1776 """process a pushkey request"""
e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1777 dec = pushkey.decode
e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1778 namespace = dec(inpart.params['namespace'])
e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1779 key = dec(inpart.params['key'])
e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1780 old = dec(inpart.params['old'])
e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1781 new = dec(inpart.params['new'])
27198 ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock Durham Goode <durham@fb.com> parents: 27175 diff changeset	1782 # Grab the transaction to ensure that we have the lock before performing the
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock Durham Goode <durham@fb.com> parents: 27175 diff changeset	1783 # pushkey.
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock Durham Goode <durham@fb.com> parents: 27175 diff changeset	1784 if op.ui.configbool('experimental', 'bundle2lazylocking'):
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock Durham Goode <durham@fb.com> parents: 27175 diff changeset	1785 op.gettransaction()
21660 e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1786 ret = op.repo.pushkey(namespace, key, old, new)
e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1787 record = {'namespace': namespace,
e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1788 'key': key,
e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1789 'old': old,
e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1790 'new': new}
e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1791 op.records.add('pushkey', record)
e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1792 if op.reply is not None:
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1793 rpart = op.reply.newpart('reply:pushkey')
34458 24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes Augie Fackler <augie@google.com> parents: 34457 diff changeset	1794 rpart.addparam(
24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes Augie Fackler <augie@google.com> parents: 34457 diff changeset	1795 'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False)
21660 e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1796 rpart.addparam('return', '%i' % ret, mandatory=False)
26017 6de96cb31176 bundle2: abort when a mandatory pushkey part fails Pierre-Yves David <pierre-yves.david@fb.com> parents: 25935 diff changeset	1797 if inpart.mandatory and not ret:
26020 a5192774e925 bundle2: introduce a PushkeyFail error to abort unbundle on pushkey error Pierre-Yves David <pierre-yves.david@fb.com> parents: 26017 diff changeset	1798 kwargs = {}
a5192774e925 bundle2: introduce a PushkeyFail error to abort unbundle on pushkey error Pierre-Yves David <pierre-yves.david@fb.com> parents: 26017 diff changeset	1799 for key in ('namespace', 'key', 'new', 'old', 'ret'):
a5192774e925 bundle2: introduce a PushkeyFail error to abort unbundle on pushkey error Pierre-Yves David <pierre-yves.david@fb.com> parents: 26017 diff changeset	1800 if key in inpart.params:
a5192774e925 bundle2: introduce a PushkeyFail error to abort unbundle on pushkey error Pierre-Yves David <pierre-yves.david@fb.com> parents: 26017 diff changeset	1801 kwargs[key] = inpart.params[key]
a5192774e925 bundle2: introduce a PushkeyFail error to abort unbundle on pushkey error Pierre-Yves David <pierre-yves.david@fb.com> parents: 26017 diff changeset	1802 raise error.PushkeyFailed(partid=str(inpart.id), **kwargs)
21660 e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1803
33814 e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1804 def _readphaseheads(inpart):
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1805 headsbyphase = [[] for i in phases.allphases]
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1806 entrysize = struct.calcsize(_fphasesentry)
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1807 while True:
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1808 entry = inpart.read(entrysize)
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1809 if len(entry) < entrysize:
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1810 if entry:
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1811 raise error.Abort(_('bad phase-heads bundle part'))
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1812 break
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1813 phase, node = struct.unpack(_fphasesentry, entry)
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1814 headsbyphase[phase].append(node)
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1815 return headsbyphase
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1816
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1817 @parthandler('phase-heads')
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1818 def handlephases(op, inpart):
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1819 """apply phases from bundle part to repo"""
e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1820 headsbyphase = _readphaseheads(inpart)
34243 57a017f79e96 phases: remove trace of addednodes in the 'phase-heads' handling Boris Feld <boris.feld@octobus.net> parents: 34191 diff changeset	1821 phases.updatephases(op.repo.unfiltered(), op.gettransaction(), headsbyphase)
34669 13dc7f29531e exchange: don't attempt phase exchange if phase-heads was in bundle Martin von Zweigbergk <martinvonz@google.com> parents: 34665 diff changeset	1822 op.records.add('phase-heads', {})
33814 e8c8d81eb864 bundle: add config option to include phases Martin von Zweigbergk <martinvonz@google.com> parents: 33813 diff changeset	1823
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1824 @parthandler('reply:pushkey', ('return', 'in-reply-to'))
21660 e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1825 def handlepushkeyreply(op, inpart):
e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1826 """retrieve the result of a pushkey request"""
e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1827 ret = int(inpart.params['return'])
e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1828 partid = int(inpart.params['in-reply-to'])
e87d2a12d41b bundle2: add ``pushkey`` support Pierre-Yves David <pierre-yves.david@fb.com> parents: 21655 diff changeset	1829 op.records.add('pushkey', {'return': ret}, partid)
22336 60786c8a2f70 bundle2: add an obsmarkers part handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21900 diff changeset	1830
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1831 @parthandler('obsmarkers')
22336 60786c8a2f70 bundle2: add an obsmarkers part handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21900 diff changeset	1832 def handleobsmarker(op, inpart):
60786c8a2f70 bundle2: add an obsmarkers part handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21900 diff changeset	1833 """add a stream of obsmarkers to the repo"""
60786c8a2f70 bundle2: add an obsmarkers part handler Pierre-Yves David <pierre-yves.david@fb.com> parents: 21900 diff changeset	1834 tr = op.gettransaction()
24912 c00e4338fa4b obsolete: experimental flag to get debug about obsmarkers exchange Pierre-Yves David <pierre-yves.david@fb.com> parents: 24825 diff changeset	1835 markerdata = inpart.read()
34282 0407a51b9d8c codemod: register core configitems using a script Jun Wu <quark@fb.com> parents: 34244 diff changeset	1836 if op.ui.config('experimental', 'obsmarkers-exchange-debug'):
24912 c00e4338fa4b obsolete: experimental flag to get debug about obsmarkers exchange Pierre-Yves David <pierre-yves.david@fb.com> parents: 24825 diff changeset	1837 op.ui.write(('obsmarker-exchange: %i bytes received\n')
c00e4338fa4b obsolete: experimental flag to get debug about obsmarkers exchange Pierre-Yves David <pierre-yves.david@fb.com> parents: 24825 diff changeset	1838 % len(markerdata))
27325 ea390d889d3a bundle2: gracefully skip 'obsmarkers' part if evolution is disabled Pierre-Yves David <pierre-yves.david@fb.com> parents: 27227 diff changeset	1839 # The mergemarkers call will crash if marker creation is not enabled.
ea390d889d3a bundle2: gracefully skip 'obsmarkers' part if evolution is disabled Pierre-Yves David <pierre-yves.david@fb.com> parents: 27227 diff changeset	1840 # we want to avoid this if the part is advisory.
ea390d889d3a bundle2: gracefully skip 'obsmarkers' part if evolution is disabled Pierre-Yves David <pierre-yves.david@fb.com> parents: 27227 diff changeset	1841 if not inpart.mandatory and op.repo.obsstore.readonly:
ea390d889d3a bundle2: gracefully skip 'obsmarkers' part if evolution is disabled Pierre-Yves David <pierre-yves.david@fb.com> parents: 27227 diff changeset	1842 op.repo.ui.debug('ignoring obsolescence markers, feature not enabled')
ea390d889d3a bundle2: gracefully skip 'obsmarkers' part if evolution is disabled Pierre-Yves David <pierre-yves.david@fb.com> parents: 27227 diff changeset	1843 return
24912 c00e4338fa4b obsolete: experimental flag to get debug about obsmarkers exchange Pierre-Yves David <pierre-yves.david@fb.com> parents: 24825 diff changeset	1844 new = op.repo.obsstore.mergemarkers(tr, markerdata)
33109 99515353c72a obsolete: invalidate "volatile" set cache after merging marker Pierre-Yves David <pierre-yves.david@octobus.net> parents: 32803 diff changeset	1845 op.repo.invalidatevolatilesets()
22337 c380fe290290 obsmarker: write a message with the number of markers added through bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 22336 diff changeset	1846 if new:
c380fe290290 obsmarker: write a message with the number of markers added through bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 22336 diff changeset	1847 op.repo.ui.status(_('%i new obsolescence markers\n') % new)
22338 8fcd56095d3b obssmarker: add a bundle2 record with the number of markers added Pierre-Yves David <pierre-yves.david@fb.com> parents: 22337 diff changeset	1848 op.records.add('obsmarkers', {'new': new})
22340 394a17de6a2d obsmarker: produce a reply part for markers received through bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 22338 diff changeset	1849 if op.reply is not None:
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1850 rpart = op.reply.newpart('reply:obsmarkers')
34458 24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes Augie Fackler <augie@google.com> parents: 34457 diff changeset	1851 rpart.addparam(
24b7c75c6441 bundle2: use bytestr() instead of str() to convert part id to bytes Augie Fackler <augie@google.com> parents: 34457 diff changeset	1852 'in-reply-to', pycompat.bytestr(inpart.id), mandatory=False)
22340 394a17de6a2d obsmarker: produce a reply part for markers received through bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 22338 diff changeset	1853 rpart.addparam('new', '%i' % new, mandatory=False)
394a17de6a2d obsmarker: produce a reply part for markers received through bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 22338 diff changeset	1854
394a17de6a2d obsmarker: produce a reply part for markers received through bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 22338 diff changeset	1855
24825 e0e28e910fa3 bundle2: rename format, parts and config to final names Pierre-Yves David <pierre-yves.david@fb.com> parents: 24787 diff changeset	1856 @parthandler('reply:obsmarkers', ('new', 'in-reply-to'))
26042 86472038dfd2 bundle2: rename duplicate handlepushkeyreply to handleobsmarkerreply Martin von Zweigbergk <martinvonz@google.com> parents: 26029 diff changeset	1857 def handleobsmarkerreply(op, inpart):
22340 394a17de6a2d obsmarker: produce a reply part for markers received through bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 22338 diff changeset	1858 """retrieve the result of a pushkey request"""
394a17de6a2d obsmarker: produce a reply part for markers received through bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 22338 diff changeset	1859 ret = int(inpart.params['new'])
394a17de6a2d obsmarker: produce a reply part for markers received through bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 22338 diff changeset	1860 partid = int(inpart.params['in-reply-to'])
394a17de6a2d obsmarker: produce a reply part for markers received through bundle2 Pierre-Yves David <pierre-yves.david@fb.com> parents: 22338 diff changeset	1861 op.records.add('obsmarkers', {'new': ret}, partid)
25935 d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1862
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1863 @parthandler('hgtagsfnodes')
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1864 def handlehgtagsfnodes(op, inpart):
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1865 """Applies .hgtags fnodes cache entries to the local repo.
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1866
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1867 Payload is pairs of 20 byte changeset nodes and filenodes.
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1868 """
27198 ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock Durham Goode <durham@fb.com> parents: 27175 diff changeset	1869 # Grab the transaction so we ensure that we have the lock at this point.
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock Durham Goode <durham@fb.com> parents: 27175 diff changeset	1870 if op.ui.configbool('experimental', 'bundle2lazylocking'):
ee1bcb9aa6e4 bundle2: add op.gettransaction() to handlers that need the lock Durham Goode <durham@fb.com> parents: 27175 diff changeset	1871 op.gettransaction()
25935 d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1872 cache = tags.hgtagsfnodescache(op.repo.unfiltered())
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1873
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1874 count = 0
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1875 while True:
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1876 node = inpart.read(20)
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1877 fnode = inpart.read(20)
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1878 if len(node) < 20 or len(fnode) < 20:
26195 c0bdfe87b245 bundle2: reword debug message for invalid .hgtags data Gregory Szorc <gregory.szorc@gmail.com> parents: 26194 diff changeset	1879 op.ui.debug('ignoring incomplete received .hgtags fnodes data\n')
25935 d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1880 break
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1881 cache.setfnode(node, fnode)
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1882 count += 1
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1883
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1884 cache.write()
d29201352af7 bundle2: part handler for processing .hgtags fnodes mappings Gregory Szorc <gregory.szorc@gmail.com> parents: 25820 diff changeset	1885 op.ui.debug('applied %i hgtags fnodes cache entries\n' % count)
34476 db3dc11356ed pushvars: move fb extension pushvars to core Pulkit Goyal <7895pulkit@gmail.com> parents: 34465 diff changeset	1886
db3dc11356ed pushvars: move fb extension pushvars to core Pulkit Goyal <7895pulkit@gmail.com> parents: 34465 diff changeset	1887 @parthandler('pushvars')
db3dc11356ed pushvars: move fb extension pushvars to core Pulkit Goyal <7895pulkit@gmail.com> parents: 34465 diff changeset	1888 def bundle2getvars(op, part):
db3dc11356ed pushvars: move fb extension pushvars to core Pulkit Goyal <7895pulkit@gmail.com> parents: 34465 diff changeset	1889 '''unbundle a bundle2 containing shellvars on the server'''
db3dc11356ed pushvars: move fb extension pushvars to core Pulkit Goyal <7895pulkit@gmail.com> parents: 34465 diff changeset	1890 # An option to disable unbundling on server-side for security reasons
34618 057d31ceace3 pushvars: add a coreconfigitem for push.pushvars.server Pulkit Goyal <7895pulkit@gmail.com> parents: 34593 diff changeset	1891 if op.ui.configbool('push', 'pushvars.server'):
34476 db3dc11356ed pushvars: move fb extension pushvars to core Pulkit Goyal <7895pulkit@gmail.com> parents: 34465 diff changeset	1892 hookargs = {}
db3dc11356ed pushvars: move fb extension pushvars to core Pulkit Goyal <7895pulkit@gmail.com> parents: 34465 diff changeset	1893 for key, value in part.advisoryparams:
db3dc11356ed pushvars: move fb extension pushvars to core Pulkit Goyal <7895pulkit@gmail.com> parents: 34465 diff changeset	1894 key = key.upper()
db3dc11356ed pushvars: move fb extension pushvars to core Pulkit Goyal <7895pulkit@gmail.com> parents: 34465 diff changeset	1895 # We want pushed variables to have USERVAR_ prepended so we know
db3dc11356ed pushvars: move fb extension pushvars to core Pulkit Goyal <7895pulkit@gmail.com> parents: 34465 diff changeset	1896 # they came from the --pushvar flag.
db3dc11356ed pushvars: move fb extension pushvars to core Pulkit Goyal <7895pulkit@gmail.com> parents: 34465 diff changeset	1897 key = "USERVAR_" + key
db3dc11356ed pushvars: move fb extension pushvars to core Pulkit Goyal <7895pulkit@gmail.com> parents: 34465 diff changeset	1898 hookargs[key] = value
db3dc11356ed pushvars: move fb extension pushvars to core Pulkit Goyal <7895pulkit@gmail.com> parents: 34465 diff changeset	1899 op.addhookargs(hookargs)

Mercurial > hg > mercurial-source

annotate mercurial/bundle2.py @ 34671:891118dcd279