How to efficiently chunk long string | Bytes (2024)

Hi All,

I'm trying to chunk a long string SourceString into lines of LineLength
using this code:

Dim sReturn As String = ""
Dim iPos As Integer = 0
Do Until iPos >= SourceString.Le ngth - LineLength
sReturn += SourceString.Su bstring(iPos, LineLength) + vbCrLf
iPos += LineLength
Loop
sReturn += SourceString.Su bstring(iPos)

It's OK if the SourceString is less than 1 mb. But if it's over 5 mb, the
cost of time and machine resource is intolerable.
Does anyone has a better idea or a routine dealing with large SourceString?
Thanks and regards,
Anony

Subscribe Reply

Anony,
To really see a difference you would need to repeat the outer loop multiple
times. A single pass over the loop may not show any significant measurable
difference...

Rather then simply timing the loop, you can use the CLR Profiler to see all
the extraneous string allocations that I was referring to.

Info on the CLR Profiler:
http://msdn.microsoft.com/library/de...nethowto13.asp

http://msdn.microsoft.com/library/de...anagedapps.asp

Hope this helps
Jay

"Anony" <An***@Nepal.NL > wrote in message
news:c8******** **@news3.tilbu1 .nb.home.nl...

Hi Cor,

I couldn't notice the difference between string + vbCrLf or one morevbCrLf in just a few sec. comparing that it was over 10 min without SB!
The allocated capacity has limited some memory usage, perhaps it increases
the speed too.

"Cor Ligthert" <no**********@p lanet.nl> wrote in message
news:%2******** *******@tk2msft ngp13.phx.gbl.. .

Hi Antony

Do you mean that changing that string + vbcrlf to 2 seperate appends,did make it again 100 times faster so that it is now 10000 times faster than
before?

What is happeng is easy, a string is immutable, which mean that every

change

in a string will result in a new string. I think that you can immaging

what

happens in the memory when the strings become longer and longer.

The stringbuilder does that not but let us say appends everytime. Thisis especially important of course with long strings.

Therefore I am suprissed that adding only vbcrlf to a string will change

the

behaviour. However I agree that when the first string is a real long

string,

that that can possible change the performance (something I think about

now),

I did test it only with short strings.

As far as I know is there nothing like the stringbuilder in VB6.

Cor

Cor,
I agree with Jon on this one!

Jay

"Cor Ligthert" <no**********@p lanet.nl> wrote in message
news:O9******** ******@TK2MSFTN GP12.phx.gbl...

Hi Jeff,

In the language.vb net group has Anony disconnected the newsgroup
microsoft.publi c.dotnet.genera l

There he has answers from not only me.

In the dotnet general group with exception from mine answer there are only
troll answers.

I hope this clears it.

Cor

Cor,
By the way, here is one of many MSDN articles on writing programs that
perform well, including using multiple Append's over string concatenation
when using the StringBuilder.

http://msdn.microsoft.com/architectu...l/scalenet.asp

The above article was published last week.
For a plethora of other articles on writing code that performs well see:

http://msdn.microsoft.com/library/de...anagedcode.asp

http://msdn.microsoft.com/library/de...anagedapps.asp

http://msdn.microsoft.com/library/de...vbnstrcatn.asp

http://msdn.microsoft.com/library/de...tchperfopt.asp

http://msdn.microsoft.com/library/de...tperftechs.asp

Hope this helps
Jay

"Cor Ligthert" <no**********@p lanet.nl> wrote in message
news:Oi******** *****@TK2MSFTNG P10.phx.gbl...

Cor,
It may be faster in a simply test.

However! You are also causing more work for the GC, so when the GCfinally needs to collect all those temporary strings, your app will slow down.

So is it really faster?

That collecting goes in null time. Therefore I think the work of the GC
should not be an isue in making a right program.

I surelly would not have stated this, because I was the first in this
newsgroup who extremly did use that code as you showed now, and recognise
with that tests that my idea about this was wrong. (Not extreme however it
was recognizable)

I myself find it still nicer code by the way with that append.

Cor

Jay,

For me was frustrating that I always send to OP's the code exactly as you
had made now, than there are others who use partially the connection in a
string. I do not start with them a discussion about that anymore.

You would have maybe seen that (it is a time ago I sand that) in my example
how to build a CSV file from a dataset using the stringbuilder, there I have
done that extremely with the stringbuilder.a ppend. However I got the idea
that nobody did understand why I did that.

I tested it some weeks ago and saw that the results from doing that was null
or almost null when it was the inline part, it seems to me as something that
plays now in the ADONET group with the datatable.selec t and the
dataview.rowfil ter, that seems that there are things done with some
invisible caching.

Therefore, I tried to tell the OP the essentials from the change in his code
and did not go for my own hobbyhorse.

Than there came a correction from you in the way as I normally always do it
as a message to the OP.

I did not understand why you did that. You could have started a discussion
in the rowe where I did give my advice to the OP as well. When you know how
I act in that case, you would have seen that I would have tested that deeper
again.

In addition, the GC is not a part of the code for me. A good program has
nothing to do with how the operating system acts today in my opinion (When
it are minor things). When you do not agree that with me, than we have in
that a different meaning, which is in my opinion no problem.

Thanks for the link I check it.

Cor

Hi Jay,

In the dotnet general group with exception from mine answer there areonly troll answers.

You mean knowing that I am a regular from the general group you do as Jon
not read it in its context as it was meant.

In the dotnet general group with exception from mine answer there are *in
this thread* only
troll answers

When you keep that, I will place a general excuse in the dotnet general
group referencing to you and Jon, because I never would insult in general
the visitors there.

Cor

Cor Ligthert <no**********@p lanet.nl> wrote:

You mean knowing that I am a regular from the general group you do as Jon
not read it in its context as it was meant.

In the dotnet general group with exception from mine answer there are *in
this thread* only troll answers
If you'd said that, I would have disagreed (pointing out that the
question is *not* a VB-specific one isn't trolling) but I wouldn't have
been as affronted.

It certainly wasn't clear that you meant that originally though. If
you'd just corrected yourself much earlier, it would have saved a lot
of hassle.
When you keep that, I will place a general excuse in the dotnet general
group referencing to you and Jon, because I never would insult in general
the visitors there.

Well that's *certainly* not how it came across, to me at least.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

Thanks Jay,

I tested a couple of rounds, each 20 passes between these two cases:

1.
SB.Append(Sourc eString.Substri ng(iPos, LineLength) + vbCrLf)

2.
SB.Append(Sourc eString.Substri ng(iPos, LineLength))
SB.Append(vbCrL f)

The SourceString is encoded, over10 Mb, LineLength = 76.
The time ticker difference for case 1 is about 32,000,000; and for case 2
about 21,000,000.
So, I'd say SB.Append(vbCrL f) is faster than + vbCrLf.

"Jay B. Harlow [MVP - Outlook]" <Ja************ @msn.com> wrote in message
news:uS******** ********@TK2MSF TNGP09.phx.gbl. ..

Anony,
To really see a difference you would need to repeat the outer loopmultiple times. A single pass over the loop may not show any significant measurable
difference...

Rather then simply timing the loop, you can use the CLR Profiler to seeall the extraneous string allocations that I was referring to.

Info on the CLR Profiler:
http://msdn.microsoft.com/library/de...nethowto13.asp
http://msdn.microsoft.com/library/de...anagedapps.asp
Hope this helps
Jay

> > You mean knowing that I am a regular from the general group you do as
Jon

not read it in its context as it was meant.

If you'd said that, I would have disagreed

Do you mean that I am not a regular from the dotnet.general newsgroup?

Hi Anony,

Thanks from me, now I do not have to test it and go back to my previous
methods, which is the append from every thing and nothing inline.

(By the way using that + is not really VB it is the &.

I was as well using the +. Now I use because Herfried did give almost
everytime a reply when I used that in my samples the &. I think most VB
people are used to it, which makes talking in this newsgroup easier when the
& is used.

Cor

Hi Jay,

I see it sit very deep in my mind, here a message I once got from someone
who had seldom had critique on me, even when it was needed. (I do not say he
was not right with this message, however it showed how much I was using it)

:-)

Hi Cor,

Sorry if I'm being a damp squib here but a StringBiulder for a single
concat (or even a few small ones) gives no advantage. It comes into its own
when it reduces the number and/or size of temporary strings by a fairly
decent
amount - such as within loops. The savings are due to reduced amounts of
copying and reduced GC activity.

Regards,
Fergus